2015-06-17 11:06:41,071 INFO [main] hbase.ResourceChecker(148): before: master.procedure.TestMasterRestartWithProcedures#testMasterProcsWithMasterRestarting Thread=4, OpenFileDescriptor=179, MaxFileDescriptor=10240, SystemLoadAverage=500, ProcessCount=254, AvailableMemoryMB=218 2015-06-17 11:06:41,075 INFO [main] hbase.HBaseTestingUtility(961): Starting up minicluster with 1 master(s) and 1 regionserver(s) and 1 datanode(s) 2015-06-17 11:06:41,097 INFO [main] hbase.HBaseTestingUtility(444): Created new mini-cluster data directory: /Users/ndimiduk/repos/hbase/hbase-server/target/test-data/31bbbe2a-d30b-406a-aae8-a42004014f71/dfscluster_728c1ca4-5c89-445c-9c45-d7f21e719166, deleteOnExit=true 2015-06-17 11:06:41,098 INFO [main] hbase.HBaseTestingUtility(691): Setting test.cache.data to /Users/ndimiduk/repos/hbase/hbase-server/target/test-data/31bbbe2a-d30b-406a-aae8-a42004014f71/cache_data in system properties and HBase conf 2015-06-17 11:06:41,098 INFO [main] hbase.HBaseTestingUtility(691): Setting hadoop.tmp.dir to /Users/ndimiduk/repos/hbase/hbase-server/target/test-data/31bbbe2a-d30b-406a-aae8-a42004014f71/hadoop_tmp in system properties and HBase conf 2015-06-17 11:06:41,098 INFO [main] hbase.HBaseTestingUtility(691): Setting hadoop.log.dir to /Users/ndimiduk/repos/hbase/hbase-server/target/test-data/31bbbe2a-d30b-406a-aae8-a42004014f71/hadoop_logs in system properties and HBase conf 2015-06-17 11:06:41,099 INFO [main] hbase.HBaseTestingUtility(691): Setting mapreduce.cluster.local.dir to /Users/ndimiduk/repos/hbase/hbase-server/target/test-data/31bbbe2a-d30b-406a-aae8-a42004014f71/mapred_local in system properties and HBase conf 2015-06-17 11:06:41,099 INFO [main] hbase.HBaseTestingUtility(691): Setting mapreduce.cluster.temp.dir to /Users/ndimiduk/repos/hbase/hbase-server/target/test-data/31bbbe2a-d30b-406a-aae8-a42004014f71/mapred_temp in system properties and HBase conf 2015-06-17 11:06:41,099 INFO [main] hbase.HBaseTestingUtility(682): read short circuit is OFF 2015-06-17 11:06:41,774 WARN [main] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2015-06-17 11:06:41,909 DEBUG [main] fs.HFileSystem(221): The file system is not a DistributedFileSystem. Skipping on block location reordering Formatting using clusterid: testClusterID 2015-06-17 11:06:42,517 WARN [main] impl.MetricsConfig(124): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2015-06-17 11:06:42,630 INFO [main] log.Slf4jLog(67): Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2015-06-17 11:06:42,673 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2015-06-17 11:06:42,690 INFO [main] log.Slf4jLog(67): Extract jar:file:/Users/ndimiduk/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.5.1/hadoop-hdfs-2.5.1-tests.jar!/webapps/hdfs to /var/folders/qj/_4zc7trx64j_9cq7dc5q1f_40000gp/T/Jetty_localhost_50894_hdfs____8aj1xe/webapp 2015-06-17 11:06:42,798 WARN [main] server.AuthenticationFilter(166): 'signature.secret' configuration not set, using a random value as secret 2015-06-17 11:06:42,825 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:50894 2015-06-17 11:06:43,310 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2015-06-17 11:06:43,317 INFO [main] log.Slf4jLog(67): Extract jar:file:/Users/ndimiduk/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.5.1/hadoop-hdfs-2.5.1-tests.jar!/webapps/datanode to /var/folders/qj/_4zc7trx64j_9cq7dc5q1f_40000gp/T/Jetty_localhost_50897_datanode____.garvzy/webapp 2015-06-17 11:06:43,399 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:50897 2015-06-17 11:06:43,719 INFO [IPC Server handler 4 on 50895] blockmanagement.BlockManager(1753): BLOCK* processReport: from storage DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d node DatanodeRegistration(127.0.0.1, datanodeUuid=96ae6757-bb52-469b-ab1e-237b0d5d1082, infoPort=50897, ipcPort=50898, storageInfo=lv=-55;cid=testClusterID;nsid=2049968863;c=0), blocks: 0, hasStaleStorages: true, processing time: 1 msecs 2015-06-17 11:06:43,719 INFO [IPC Server handler 4 on 50895] blockmanagement.BlockManager(1753): BLOCK* processReport: from storage DS-68dc2820-54ff-46e7-8182-20d6562bff0e node DatanodeRegistration(127.0.0.1, datanodeUuid=96ae6757-bb52-469b-ab1e-237b0d5d1082, infoPort=50897, ipcPort=50898, storageInfo=lv=-55;cid=testClusterID;nsid=2049968863;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs 2015-06-17 11:06:43,781 INFO [main] zookeeper.MiniZooKeeperCluster(273): Started MiniZooKeeperCluster and ran successful 'stat' on client port=51848 2015-06-17 11:06:43,801 INFO [main] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:06:43,804 INFO [main] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:06:43,976 INFO [IPC Server handler 1 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 0 2015-06-17 11:06:43,988 INFO [main] util.FSUtils(751): Created version file at hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034 with version=8 2015-06-17 11:06:44,281 INFO [main] client.ConnectionUtils(105): master//10.0.0.110:0 server-side HConnection retries=350 2015-06-17 11:06:44,451 INFO [main] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2015-06-17 11:06:44,463 INFO [main] ipc.RpcServer$Listener(598): master//10.0.0.110:0: started 10 reader(s). 2015-06-17 11:06:44,500 INFO [main] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:06:44,547 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=master:50903 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:06:44,579 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): master:509030x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:06:44,584 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(578): master:50903-0x14e02b3a2320000 connected 2015-06-17 11:06:44,615 DEBUG [main] zookeeper.ZKUtil(488): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:06:44,627 DEBUG [main] zookeeper.ZKUtil(488): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2015-06-17 11:06:44,633 INFO [RpcServer.responder] ipc.RpcServer$Responder(917): RpcServer.responder: starting 2015-06-17 11:06:44,633 INFO [RpcServer.listener,port=50903] ipc.RpcServer$Listener(729): RpcServer.listener,port=50903: starting 2015-06-17 11:06:44,634 DEBUG [main] ipc.RpcExecutor(114): B.default Start Handler index=0 queue=0 2015-06-17 11:06:44,634 DEBUG [main] ipc.RpcExecutor(114): B.default Start Handler index=1 queue=0 2015-06-17 11:06:44,634 DEBUG [main] ipc.RpcExecutor(114): B.default Start Handler index=2 queue=0 2015-06-17 11:06:44,635 DEBUG [main] ipc.RpcExecutor(114): B.default Start Handler index=3 queue=0 2015-06-17 11:06:44,635 DEBUG [main] ipc.RpcExecutor(114): B.default Start Handler index=4 queue=0 2015-06-17 11:06:44,635 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=0 queue=0 2015-06-17 11:06:44,636 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=1 queue=1 2015-06-17 11:06:44,636 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=2 queue=0 2015-06-17 11:06:44,636 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=3 queue=1 2015-06-17 11:06:44,636 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=4 queue=0 2015-06-17 11:06:44,637 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=5 queue=1 2015-06-17 11:06:44,637 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=6 queue=0 2015-06-17 11:06:44,637 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=7 queue=1 2015-06-17 11:06:44,637 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=8 queue=0 2015-06-17 11:06:44,638 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=9 queue=1 2015-06-17 11:06:44,638 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=10 queue=0 2015-06-17 11:06:44,638 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=11 queue=1 2015-06-17 11:06:44,638 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=12 queue=0 2015-06-17 11:06:44,639 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=13 queue=1 2015-06-17 11:06:44,639 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=14 queue=0 2015-06-17 11:06:44,639 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=15 queue=1 2015-06-17 11:06:44,639 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=16 queue=0 2015-06-17 11:06:44,640 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=17 queue=1 2015-06-17 11:06:44,640 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=18 queue=0 2015-06-17 11:06:44,640 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=19 queue=1 2015-06-17 11:06:44,640 DEBUG [main] ipc.RpcExecutor(114): Replication Start Handler index=0 queue=0 2015-06-17 11:06:44,641 DEBUG [main] ipc.RpcExecutor(114): Replication Start Handler index=1 queue=0 2015-06-17 11:06:44,641 DEBUG [main] ipc.RpcExecutor(114): Replication Start Handler index=2 queue=0 2015-06-17 11:06:44,645 INFO [main] master.HMaster(349): hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034, hbase.cluster.distributed=false 2015-06-17 11:06:44,658 INFO [main] master.HMaster(1512): Adding backup master ZNode /hbase/backup-masters/10.0.0.110,50903,1434564404489 2015-06-17 11:06:44,666 DEBUG [main] zookeeper.ZKUtil(486): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/10.0.0.110,50903,1434564404489 2015-06-17 11:06:44,679 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:06:44,680 DEBUG [10.0.0.110:50903.activeMasterManager] zookeeper.ZKUtil(486): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:06:44,681 INFO [10.0.0.110:50903.activeMasterManager] master.ActiveMasterManager(170): Deleting ZNode for /hbase/backup-masters/10.0.0.110,50903,1434564404489 from backup master directory 2015-06-17 11:06:44,683 DEBUG [main-EventThread] zookeeper.ZKUtil(486): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:06:44,683 DEBUG [main-EventThread] master.ActiveMasterManager(126): A master is now available 2015-06-17 11:06:44,683 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/10.0.0.110,50903,1434564404489 2015-06-17 11:06:44,684 WARN [10.0.0.110:50903.activeMasterManager] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:06:44,684 INFO [10.0.0.110:50903.activeMasterManager] master.ActiveMasterManager(179): Registered Active Master=10.0.0.110,50903,1434564404489 2015-06-17 11:06:44,759 INFO [main] client.ConnectionUtils(105): regionserver//10.0.0.110:0 server-side HConnection retries=350 2015-06-17 11:06:44,760 INFO [main] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2015-06-17 11:06:44,762 INFO [main] ipc.RpcServer$Listener(598): regionserver//10.0.0.110:0: started 10 reader(s). 2015-06-17 11:06:44,766 INFO [main] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:06:44,768 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=regionserver:50905 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:06:44,771 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:509050x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:06:44,772 DEBUG [main] zookeeper.ZKUtil(486): regionserver:509050x0, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:06:44,773 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(578): regionserver:50905-0x14e02b3a2320001 connected 2015-06-17 11:06:44,774 DEBUG [main] zookeeper.ZKUtil(488): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2015-06-17 11:06:44,774 INFO [RpcServer.responder] ipc.RpcServer$Responder(917): RpcServer.responder: starting 2015-06-17 11:06:44,774 INFO [RpcServer.listener,port=50905] ipc.RpcServer$Listener(729): RpcServer.listener,port=50905: starting 2015-06-17 11:06:44,774 DEBUG [main] ipc.RpcExecutor(114): B.default Start Handler index=0 queue=0 2015-06-17 11:06:44,775 DEBUG [main] ipc.RpcExecutor(114): B.default Start Handler index=1 queue=0 2015-06-17 11:06:44,775 DEBUG [main] ipc.RpcExecutor(114): B.default Start Handler index=2 queue=0 2015-06-17 11:06:44,775 DEBUG [main] ipc.RpcExecutor(114): B.default Start Handler index=3 queue=0 2015-06-17 11:06:44,776 DEBUG [main] ipc.RpcExecutor(114): B.default Start Handler index=4 queue=0 2015-06-17 11:06:44,776 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=0 queue=0 2015-06-17 11:06:44,776 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=1 queue=1 2015-06-17 11:06:44,778 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=2 queue=0 2015-06-17 11:06:44,779 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=3 queue=1 2015-06-17 11:06:44,779 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=4 queue=0 2015-06-17 11:06:44,779 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=5 queue=1 2015-06-17 11:06:44,780 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=6 queue=0 2015-06-17 11:06:44,780 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=7 queue=1 2015-06-17 11:06:44,780 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=8 queue=0 2015-06-17 11:06:44,780 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=9 queue=1 2015-06-17 11:06:44,781 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=10 queue=0 2015-06-17 11:06:44,781 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=11 queue=1 2015-06-17 11:06:44,781 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=12 queue=0 2015-06-17 11:06:44,781 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=13 queue=1 2015-06-17 11:06:44,781 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=14 queue=0 2015-06-17 11:06:44,782 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=15 queue=1 2015-06-17 11:06:44,782 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=16 queue=0 2015-06-17 11:06:44,782 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=17 queue=1 2015-06-17 11:06:44,782 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=18 queue=0 2015-06-17 11:06:44,783 DEBUG [main] ipc.RpcExecutor(114): Priority Start Handler index=19 queue=1 2015-06-17 11:06:44,783 DEBUG [main] ipc.RpcExecutor(114): Replication Start Handler index=0 queue=0 2015-06-17 11:06:44,783 DEBUG [main] ipc.RpcExecutor(114): Replication Start Handler index=1 queue=0 2015-06-17 11:06:44,783 DEBUG [main] ipc.RpcExecutor(114): Replication Start Handler index=2 queue=0 2015-06-17 11:06:44,790 INFO [main] http.HttpRequestLog(69): Http request log for http.requests.regionserver is not defined 2015-06-17 11:06:44,792 INFO [IPC Server handler 4 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741826_1002{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 0 2015-06-17 11:06:44,792 INFO [main] http.HttpServer(817): Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2015-06-17 11:06:44,793 INFO [main] http.HttpServer(795): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context regionserver 2015-06-17 11:06:44,793 INFO [main] http.HttpServer(802): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2015-06-17 11:06:44,794 INFO [main] http.HttpServer(802): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2015-06-17 11:06:44,794 DEBUG [10.0.0.110:50903.activeMasterManager] util.FSUtils(903): Created cluster ID file at hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/hbase.id with ID: b7c4b21b-aab2-44b4-8562-2b61af22ad6c 2015-06-17 11:06:44,799 INFO [main] http.HttpServer(1009): Jetty bound to port 50909 2015-06-17 11:06:44,799 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2015-06-17 11:06:44,819 INFO [10.0.0.110:50903.activeMasterManager] master.MasterFileSystem(532): BOOTSTRAP: creating hbase:meta region 2015-06-17 11:06:44,821 INFO [10.0.0.110:50903.activeMasterManager] regionserver.HRegion(5888): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '3', TTL => 'FOREVER', MIN_VERSIONS => '0', CACHE_DATA_IN_L1 => 'true', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '8192', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'table', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '10', TTL => 'FOREVER', MIN_VERSIONS => '0', CACHE_DATA_IN_L1 => 'true', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '8192', IN_MEMORY => 'true', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034 Table name == hbase:meta 2015-06-17 11:06:44,826 INFO [main] log.Slf4jLog(67): Started SelectChannelConnector@0.0.0.0:50909 2015-06-17 11:06:44,842 INFO [IPC Server handler 6 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741827_1003{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 0 2015-06-17 11:06:44,846 DEBUG [10.0.0.110:50903.activeMasterManager] regionserver.HRegion(722): Instantiated hbase:meta,,1.1588230740 2015-06-17 11:06:44,849 INFO [M:0;10.0.0.110:50903] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x6c6010c7 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:06:44,849 INFO [RS:0;10.0.0.110:50905] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x6c6010c7 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:06:44,852 DEBUG [RS:0;10.0.0.110:50905-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x6c6010c70x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:06:44,853 DEBUG [M:0;10.0.0.110:50903-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x6c6010c70x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:06:44,853 INFO [RS:0;10.0.0.110:50905] client.ZooKeeperRegistry(104): ClusterId read in ZooKeeper is null 2015-06-17 11:06:44,853 DEBUG [RS:0;10.0.0.110:50905] client.ConnectionImplementation(481): clusterid came back null, using default default-cluster 2015-06-17 11:06:44,853 INFO [M:0;10.0.0.110:50903] client.ZooKeeperRegistry(104): ClusterId read in ZooKeeper is null 2015-06-17 11:06:44,853 DEBUG [M:0;10.0.0.110:50903] client.ConnectionImplementation(481): clusterid came back null, using default default-cluster 2015-06-17 11:06:44,854 DEBUG [RS:0;10.0.0.110:50905-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x6c6010c7-0x14e02b3a2320002 connected 2015-06-17 11:06:44,854 DEBUG [M:0;10.0.0.110:50903-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x6c6010c7-0x14e02b3a2320003 connected 2015-06-17 11:06:44,886 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(488): Allocating LruBlockCache size=995.60 MB, blockSize=64 KB 2015-06-17 11:06:44,887 DEBUG [RS:0;10.0.0.110:50905] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68662128, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:06:44,887 DEBUG [M:0;10.0.0.110:50903] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@90afea4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:06:44,887 DEBUG [RS:0;10.0.0.110:50905] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:06:44,888 DEBUG [M:0;10.0.0.110:50903] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:06:44,888 DEBUG [RS:0;10.0.0.110:50905] ipc.AsyncRpcClient(136): Create NioEventLoopGroup with maxThreads = 0 2015-06-17 11:06:44,895 DEBUG [StoreOpener-1588230740-1] hfile.CacheConfig(502): Trying to use Internal l2 cache 2015-06-17 11:06:44,895 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:06:44,901 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:06:44,904 DEBUG [StoreOpener-1588230740-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info 2015-06-17 11:06:44,908 DEBUG [RS:0;10.0.0.110:50905] ipc.AsyncRpcClient(111): Create global event loop group NioEventLoopGroup 2015-06-17 11:06:44,909 DEBUG [RS:0;10.0.0.110:50905] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:06:44,909 DEBUG [M:0;10.0.0.110:50903] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:06:44,911 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:06:44,912 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:06:44,914 DEBUG [StoreOpener-1588230740-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table 2015-06-17 11:06:44,921 DEBUG [10.0.0.110:50903.activeMasterManager] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740 2015-06-17 11:06:44,930 DEBUG [10.0.0.110:50903.activeMasterManager] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:06:44,935 DEBUG [10.0.0.110:50903.activeMasterManager] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-06-17 11:06:44,935 INFO [10.0.0.110:50903.activeMasterManager] regionserver.HRegion(848): Onlined 1588230740; next sequenceid=2 2015-06-17 11:06:44,936 DEBUG [10.0.0.110:50903.activeMasterManager] regionserver.HRegion(1365): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2015-06-17 11:06:44,936 DEBUG [10.0.0.110:50903.activeMasterManager] regionserver.HRegion(1392): Updates disabled for region hbase:meta,,1.1588230740 2015-06-17 11:06:44,937 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(870): Closed info 2015-06-17 11:06:44,937 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(870): Closed table 2015-06-17 11:06:44,937 INFO [10.0.0.110:50903.activeMasterManager] regionserver.HRegion(1486): Closed hbase:meta,,1.1588230740 2015-06-17 11:06:44,964 INFO [IPC Server handler 8 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741828_1004{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 0 2015-06-17 11:06:44,967 DEBUG [10.0.0.110:50903.activeMasterManager] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2015-06-17 11:06:44,978 INFO [10.0.0.110:50903.activeMasterManager] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:06:44,981 DEBUG [10.0.0.110:50903.activeMasterManager] coordination.ZKSplitLogManagerCoordination(885): Distributed log replay=true 2015-06-17 11:06:44,984 INFO [10.0.0.110:50903.activeMasterManager] coordination.ZKSplitLogManagerCoordination(599): Found 0 orphan tasks and 0 rescan nodes 2015-06-17 11:06:44,984 DEBUG [10.0.0.110:50903.activeMasterManager] util.FSTableDescriptors(222): Fetching table descriptors from the filesystem. 2015-06-17 11:06:44,995 INFO [10.0.0.110:50903.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x499c9a7 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:06:44,998 DEBUG [10.0.0.110:50903.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x499c9a70x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:06:44,999 DEBUG [10.0.0.110:50903.activeMasterManager] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3fc5b8b9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:06:44,999 DEBUG [10.0.0.110:50903.activeMasterManager] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:06:44,999 DEBUG [10.0.0.110:50903.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x499c9a7-0x14e02b3a2320004 connected 2015-06-17 11:06:44,999 DEBUG [10.0.0.110:50903.activeMasterManager] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:06:45,011 INFO [10.0.0.110:50903.activeMasterManager] balancer.StochasticLoadBalancer(135): loading config 2015-06-17 11:06:45,021 DEBUG [10.0.0.110:50903.activeMasterManager] zookeeper.ZKUtil(488): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2015-06-17 11:06:45,045 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2015-06-17 11:06:45,045 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2015-06-17 11:06:45,046 INFO [10.0.0.110:50903.activeMasterManager] master.HMaster(568): Server active/primary master=10.0.0.110,50903,1434564404489, sessionid=0x14e02b3a2320000, setting cluster-up flag (Was=false) 2015-06-17 11:06:45,047 INFO [M:0;10.0.0.110:50903] regionserver.HRegionServer(774): ClusterId : b7c4b21b-aab2-44b4-8562-2b61af22ad6c 2015-06-17 11:06:45,047 INFO [RS:0;10.0.0.110:50905] regionserver.HRegionServer(774): ClusterId : b7c4b21b-aab2-44b4-8562-2b61af22ad6c 2015-06-17 11:06:45,052 DEBUG [RS:0;10.0.0.110:50905] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2015-06-17 11:06:45,052 DEBUG [M:0;10.0.0.110:50903] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2015-06-17 11:06:45,056 DEBUG [RS:0;10.0.0.110:50905] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot already exists 2015-06-17 11:06:45,057 DEBUG [RS:0;10.0.0.110:50905] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:06:45,059 DEBUG [RS:0;10.0.0.110:50905] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/abort already exists 2015-06-17 11:06:45,059 DEBUG [10.0.0.110:50903.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:06:45,060 DEBUG [RS:0;10.0.0.110:50905] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2015-06-17 11:06:45,060 DEBUG [M:0;10.0.0.110:50903] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2015-06-17 11:06:45,060 DEBUG [M:0;10.0.0.110:50903] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2015-06-17 11:06:45,060 DEBUG [RS:0;10.0.0.110:50905] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2015-06-17 11:06:45,060 INFO [10.0.0.110:50903.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2015-06-17 11:06:45,062 DEBUG [RS:0;10.0.0.110:50905] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc already exists 2015-06-17 11:06:45,063 DEBUG [10.0.0.110:50903.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,50903,1434564404489 2015-06-17 11:06:45,063 DEBUG [RS:0;10.0.0.110:50905] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:06:45,064 DEBUG [RS:0;10.0.0.110:50905] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/reached already exists 2015-06-17 11:06:45,065 DEBUG [RS:0;10.0.0.110:50905] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/abort already exists 2015-06-17 11:06:45,065 DEBUG [M:0;10.0.0.110:50903] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2015-06-17 11:06:45,065 DEBUG [10.0.0.110:50903.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:06:45,065 DEBUG [RS:0;10.0.0.110:50905] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2015-06-17 11:06:45,067 INFO [10.0.0.110:50903.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2015-06-17 11:06:45,068 DEBUG [10.0.0.110:50903.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,50903,1434564404489 2015-06-17 11:06:45,068 INFO [M:0;10.0.0.110:50903] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2015-06-17 11:06:45,068 INFO [RS:0;10.0.0.110:50905] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2015-06-17 11:06:45,075 INFO [RS:0;10.0.0.110:50905] compactions.PressureAwareCompactionThroughputController(175): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2015-06-17 11:06:45,075 INFO [M:0;10.0.0.110:50903] compactions.PressureAwareCompactionThroughputController(175): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2015-06-17 11:06:45,076 INFO [RS:0;10.0.0.110:50905] regionserver.HRegionServer$CompactionChecker(1504): CompactionChecker runs every 1sec 2015-06-17 11:06:45,076 INFO [M:0;10.0.0.110:50903] regionserver.HRegionServer$CompactionChecker(1504): CompactionChecker runs every 1sec 2015-06-17 11:06:45,078 DEBUG [M:0;10.0.0.110:50903] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15e9fad8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=10.0.0.110/10.0.0.110:0 2015-06-17 11:06:45,078 DEBUG [M:0;10.0.0.110:50903] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:06:45,078 DEBUG [RS:0;10.0.0.110:50905] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34538625, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=10.0.0.110/10.0.0.110:0 2015-06-17 11:06:45,078 DEBUG [RS:0;10.0.0.110:50905] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:06:45,078 DEBUG [M:0;10.0.0.110:50903] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:06:45,079 DEBUG [RS:0;10.0.0.110:50905] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:06:45,082 DEBUG [M:0;10.0.0.110:50903] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:M:0;10.0.0.110:50903 2015-06-17 11:06:45,082 DEBUG [RS:0;10.0.0.110:50905] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:RS:0;10.0.0.110:50905 2015-06-17 11:06:45,087 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:06:45,087 DEBUG [M:0;10.0.0.110:50903] zookeeper.ZKUtil(486): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50903,1434564404489 2015-06-17 11:06:45,088 DEBUG [RS:0;10.0.0.110:50905] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:06:45,088 DEBUG [main-EventThread] zookeeper.ZKUtil(486): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:06:45,088 DEBUG [main-EventThread] zookeeper.ZKUtil(486): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50903,1434564404489 2015-06-17 11:06:45,089 DEBUG [main-EventThread] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:06:45,090 DEBUG [main-EventThread] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,50903,1434564404489 2015-06-17 11:06:45,094 INFO [M:0;10.0.0.110:50903] regionserver.RegionServerCoprocessorHost(65): System coprocessor loading is enabled 2015-06-17 11:06:45,094 INFO [RS:0;10.0.0.110:50905] regionserver.RegionServerCoprocessorHost(65): System coprocessor loading is enabled 2015-06-17 11:06:45,094 INFO [M:0;10.0.0.110:50903] regionserver.RegionServerCoprocessorHost(66): Table coprocessor loading is enabled 2015-06-17 11:06:45,095 INFO [RS:0;10.0.0.110:50905] regionserver.RegionServerCoprocessorHost(66): Table coprocessor loading is enabled 2015-06-17 11:06:45,095 INFO [M:0;10.0.0.110:50903] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,50903,1434564404489 with port=50903, startcode=1434564404489 2015-06-17 11:06:45,096 INFO [RS:0;10.0.0.110:50905] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,50903,1434564404489 with port=50905, startcode=1434564404764 2015-06-17 11:06:45,097 DEBUG [M:0;10.0.0.110:50903] regionserver.HRegionServer(2295): Master is not running yet 2015-06-17 11:06:45,097 WARN [M:0;10.0.0.110:50903] regionserver.HRegionServer(900): reportForDuty failed; sleeping and then retrying. 2015-06-17 11:06:45,102 INFO [10.0.0.110:50903.activeMasterManager] master.MasterCoprocessorHost(86): System coprocessor loading is enabled 2015-06-17 11:06:45,103 DEBUG [10.0.0.110:50903.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_OPEN_REGION-10.0.0.110:50903, corePoolSize=5, maxPoolSize=5 2015-06-17 11:06:45,104 DEBUG [10.0.0.110:50903.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_CLOSE_REGION-10.0.0.110:50903, corePoolSize=5, maxPoolSize=5 2015-06-17 11:06:45,104 DEBUG [10.0.0.110:50903.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_SERVER_OPERATIONS-10.0.0.110:50903, corePoolSize=5, maxPoolSize=5 2015-06-17 11:06:45,104 DEBUG [10.0.0.110:50903.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_META_SERVER_OPERATIONS-10.0.0.110:50903, corePoolSize=5, maxPoolSize=5 2015-06-17 11:06:45,104 DEBUG [10.0.0.110:50903.activeMasterManager] executor.ExecutorService(91): Starting executor service name=M_LOG_REPLAY_OPS-10.0.0.110:50903, corePoolSize=10, maxPoolSize=10 2015-06-17 11:06:45,104 DEBUG [10.0.0.110:50903.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_TABLE_OPERATIONS-10.0.0.110:50903, corePoolSize=1, maxPoolSize=1 2015-06-17 11:06:45,115 INFO [10.0.0.110:50903.activeMasterManager] procedure2.ProcedureExecutor(446): Starting procedure executor threads=9 2015-06-17 11:06:45,115 INFO [10.0.0.110:50903.activeMasterManager] wal.WALProcedureStore(214): Starting WAL Procedure Store lease recovery 2015-06-17 11:06:45,117 WARN [10.0.0.110:50903.activeMasterManager] wal.WALProcedureStore(703): Log directory not found: File hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs does not exist. 2015-06-17 11:06:45,125 DEBUG [10.0.0.110:50903.activeMasterManager] wal.WALProcedureStore(610): Roll new state log: 1 2015-06-17 11:06:45,126 INFO [10.0.0.110:50903.activeMasterManager] wal.WALProcedureStore(237): Lease acquired for flushLogId: 1 2015-06-17 11:06:45,127 DEBUG [10.0.0.110:50903.activeMasterManager] wal.WALProcedureStore(251): No state logs to replay. 2015-06-17 11:06:45,127 DEBUG [10.0.0.110:50903.activeMasterManager] procedure2.ProcedureExecutor$1(278): load procedures maxProcId=0 2015-06-17 11:06:45,132 DEBUG [10.0.0.110:50903.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2015-06-17 11:06:45,133 INFO [10.0.0.110:50903.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:06:45,135 DEBUG [10.0.0.110:50903.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:06:45,137 DEBUG [10.0.0.110:50903.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): replicationLogCleaner-0x14e02b3a2320005 connected 2015-06-17 11:06:45,143 DEBUG [10.0.0.110:50903.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2015-06-17 11:06:45,146 DEBUG [10.0.0.110:50903.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner 2015-06-17 11:06:45,147 DEBUG [10.0.0.110:50903.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2015-06-17 11:06:45,148 DEBUG [10.0.0.110:50903.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2015-06-17 11:06:45,148 DEBUG [10.0.0.110:50903.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2015-06-17 11:06:45,148 INFO [10.0.0.110:50903.activeMasterManager] master.ServerManager(1011): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 1, timeout of 4500 ms, interval of 1500 ms. 2015-06-17 11:06:45,148 INFO [M:0;10.0.0.110:50903] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,50903,1434564404489 with port=50903, startcode=1434564404489 2015-06-17 11:06:45,149 DEBUG [RpcServer.listener,port=50903] ipc.RpcServer$Listener(840): RpcServer.listener,port=50903: connection from 10.0.0.110:50916; # active connections: 1 2015-06-17 11:06:45,159 INFO [M:0;10.0.0.110:50903] master.ServerManager(459): Registering server=10.0.0.110,50903,1434564404489 2015-06-17 11:06:45,162 INFO [M:0;10.0.0.110:50903] regionserver.HRegionServer(1341): Config from master: hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034 2015-06-17 11:06:45,163 INFO [M:0;10.0.0.110:50903] regionserver.HRegionServer(1341): Config from master: fs.defaultFS=hdfs://localhost:50895 2015-06-17 11:06:45,163 INFO [M:0;10.0.0.110:50903] regionserver.HRegionServer(1341): Config from master: hbase.master.info.port=-1 2015-06-17 11:06:45,163 WARN [M:0;10.0.0.110:50903] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:06:45,163 INFO [M:0;10.0.0.110:50903] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:06:45,164 DEBUG [M:0;10.0.0.110:50903] regionserver.HRegionServer(1603): logdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50903,1434564404489 2015-06-17 11:06:45,172 DEBUG [AsyncRpcChannel-pool2-t1] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2015-06-17 11:06:45,175 DEBUG [M:0;10.0.0.110:50903] regionserver.Replication(142): ReplicationStatisticsThread 300 2015-06-17 11:06:45,183 INFO [M:0;10.0.0.110:50903] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:06:45,195 INFO [M:0;10.0.0.110:50903] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.0.0.110%2C50903%2C1434564404489.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50903,1434564404489, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:06:45,199 INFO [10.0.0.110:50903.activeMasterManager] master.ServerManager(1028): Finished waiting for region servers count to settle; checked in 1, slept for 51 ms, expecting minimum of 1, maximum of 1, master is running 2015-06-17 11:06:45,200 INFO [10.0.0.110:50903.activeMasterManager] master.ServerManager(459): Registering server=10.0.0.110,50905,1434564404764 2015-06-17 11:06:45,200 INFO [10.0.0.110:50903.activeMasterManager] master.HMaster(665): Registered server found up in zk but who has not yet reported in: 10.0.0.110,50905,1434564404764 2015-06-17 11:06:45,203 INFO [10.0.0.110:50903.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50903,1434564404489 belongs to an existing region server 2015-06-17 11:06:45,204 INFO [RpcServer.reader=1,bindAddress=10.0.0.110,port=50903] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 50916 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:06:45,205 DEBUG [10.0.0.110:50903.activeMasterManager] zookeeper.ZKUtil(745): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2015-06-17 11:06:45,212 INFO [PriorityRpcServer.handler=0,queue=0,port=50903] master.ServerManager(459): Registering server=10.0.0.110,50905,1434564404764 2015-06-17 11:06:45,217 INFO [M:0;10.0.0.110:50903] wal.FSHLog(1365): Slow sync cost: 6 ms, current pipeline: [] 2015-06-17 11:06:45,217 INFO [M:0;10.0.0.110:50903] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50903,1434564404489/10.0.0.110%2C50903%2C1434564404489.default.1434564405195 2015-06-17 11:06:45,222 INFO [RS:0;10.0.0.110:50905] regionserver.HRegionServer(1341): Config from master: hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034 2015-06-17 11:06:45,223 INFO [RS:0;10.0.0.110:50905] regionserver.HRegionServer(1341): Config from master: fs.defaultFS=hdfs://localhost:50895 2015-06-17 11:06:45,223 INFO [RS:0;10.0.0.110:50905] regionserver.HRegionServer(1341): Config from master: hbase.master.info.port=-1 2015-06-17 11:06:45,223 WARN [RS:0;10.0.0.110:50905] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:06:45,223 INFO [RS:0;10.0.0.110:50905] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:06:45,223 DEBUG [RS:0;10.0.0.110:50905] regionserver.HRegionServer(1603): logdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50905,1434564404764 2015-06-17 11:06:45,228 DEBUG [RS:0;10.0.0.110:50905] regionserver.Replication(142): ReplicationStatisticsThread 300 2015-06-17 11:06:45,229 INFO [RS:0;10.0.0.110:50905] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:06:45,234 INFO [RS:0;10.0.0.110:50905] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.0.0.110%2C50905%2C1434564404764.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50905,1434564404764, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:06:45,244 INFO [RS:0;10.0.0.110:50905] wal.FSHLog(1365): Slow sync cost: 6 ms, current pipeline: [] 2015-06-17 11:06:45,244 INFO [RS:0;10.0.0.110:50905] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50905,1434564404764/10.0.0.110%2C50905%2C1434564404764.default.1434564405234 2015-06-17 11:06:45,250 INFO [M:0;10.0.0.110:50903] regionserver.MetricsRegionServerWrapperImpl(113): Computing regionserver metrics every 5000 milliseconds 2015-06-17 11:06:45,250 INFO [RS:0;10.0.0.110:50905] regionserver.MetricsRegionServerWrapperImpl(113): Computing regionserver metrics every 5000 milliseconds 2015-06-17 11:06:45,256 DEBUG [M:0;10.0.0.110:50903] executor.ExecutorService(91): Starting executor service name=RS_OPEN_REGION-10.0.0.110:50903, corePoolSize=3, maxPoolSize=3 2015-06-17 11:06:45,256 DEBUG [M:0;10.0.0.110:50903] executor.ExecutorService(91): Starting executor service name=RS_OPEN_META-10.0.0.110:50903, corePoolSize=1, maxPoolSize=1 2015-06-17 11:06:45,256 DEBUG [M:0;10.0.0.110:50903] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_REGION-10.0.0.110:50903, corePoolSize=3, maxPoolSize=3 2015-06-17 11:06:45,256 DEBUG [RS:0;10.0.0.110:50905] executor.ExecutorService(91): Starting executor service name=RS_OPEN_REGION-10.0.0.110:50905, corePoolSize=3, maxPoolSize=3 2015-06-17 11:06:45,256 DEBUG [M:0;10.0.0.110:50903] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_META-10.0.0.110:50903, corePoolSize=1, maxPoolSize=1 2015-06-17 11:06:45,256 DEBUG [RS:0;10.0.0.110:50905] executor.ExecutorService(91): Starting executor service name=RS_OPEN_META-10.0.0.110:50905, corePoolSize=1, maxPoolSize=1 2015-06-17 11:06:45,257 DEBUG [M:0;10.0.0.110:50903] executor.ExecutorService(91): Starting executor service name=RS_LOG_REPLAY_OPS-10.0.0.110:50903, corePoolSize=2, maxPoolSize=2 2015-06-17 11:06:45,257 DEBUG [RS:0;10.0.0.110:50905] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_REGION-10.0.0.110:50905, corePoolSize=3, maxPoolSize=3 2015-06-17 11:06:45,257 DEBUG [M:0;10.0.0.110:50903] executor.ExecutorService(91): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-10.0.0.110:50903, corePoolSize=3, maxPoolSize=3 2015-06-17 11:06:45,257 DEBUG [RS:0;10.0.0.110:50905] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_META-10.0.0.110:50905, corePoolSize=1, maxPoolSize=1 2015-06-17 11:06:45,257 DEBUG [RS:0;10.0.0.110:50905] executor.ExecutorService(91): Starting executor service name=RS_LOG_REPLAY_OPS-10.0.0.110:50905, corePoolSize=2, maxPoolSize=2 2015-06-17 11:06:45,257 DEBUG [RS:0;10.0.0.110:50905] executor.ExecutorService(91): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-10.0.0.110:50905, corePoolSize=3, maxPoolSize=3 2015-06-17 11:06:45,259 DEBUG [M:0;10.0.0.110:50903] zookeeper.ZKUtil(486): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:06:45,259 DEBUG [RS:0;10.0.0.110:50905] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:06:45,260 DEBUG [M:0;10.0.0.110:50903] zookeeper.ZKUtil(486): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50903,1434564404489 2015-06-17 11:06:45,260 INFO [M:0;10.0.0.110:50903] regionserver.ReplicationSourceManager(229): Current list of replicators: [10.0.0.110,50905,1434564404764, 10.0.0.110,50903,1434564404489] other RSs: [10.0.0.110,50905,1434564404764, 10.0.0.110,50903,1434564404489] 2015-06-17 11:06:45,260 DEBUG [RS:0;10.0.0.110:50905] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50903,1434564404489 2015-06-17 11:06:45,260 INFO [RS:0;10.0.0.110:50905] regionserver.ReplicationSourceManager(229): Current list of replicators: [10.0.0.110,50905,1434564404764, 10.0.0.110,50903,1434564404489] other RSs: [10.0.0.110,50905,1434564404764, 10.0.0.110,50903,1434564404489] 2015-06-17 11:06:45,287 INFO [M:0;10.0.0.110:50903] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x1522864c connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:06:45,287 INFO [RS:0;10.0.0.110:50905] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0xd9ef26b connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:06:45,290 DEBUG [RS:0;10.0.0.110:50905-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0xd9ef26b0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:06:45,290 DEBUG [M:0;10.0.0.110:50903-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x1522864c0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:06:45,291 DEBUG [RS:0;10.0.0.110:50905-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0xd9ef26b-0x14e02b3a2320006 connected 2015-06-17 11:06:45,292 DEBUG [M:0;10.0.0.110:50903-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x1522864c-0x14e02b3a2320007 connected 2015-06-17 11:06:45,292 DEBUG [RS:0;10.0.0.110:50905] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodecWithTags@653bf4a6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:06:45,292 DEBUG [M:0;10.0.0.110:50903] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodecWithTags@1f4a7ea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:06:45,292 DEBUG [M:0;10.0.0.110:50903] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:06:45,292 DEBUG [RS:0;10.0.0.110:50905] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:06:45,292 DEBUG [M:0;10.0.0.110:50903] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:06:45,292 DEBUG [RS:0;10.0.0.110:50905] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:06:45,323 INFO [SplitLogWorker-10.0.0.110:50903] regionserver.SplitLogWorker(134): SplitLogWorker 10.0.0.110,50903,1434564404489 starting 2015-06-17 11:06:45,324 INFO [SplitLogWorker-10.0.0.110:50905] regionserver.SplitLogWorker(134): SplitLogWorker 10.0.0.110,50905,1434564404764 starting 2015-06-17 11:06:45,325 INFO [M:0;10.0.0.110:50903] regionserver.HeapMemoryManager(190): Starting HeapMemoryTuner chore. 2015-06-17 11:06:45,325 INFO [RS:0;10.0.0.110:50905] regionserver.HeapMemoryManager(190): Starting HeapMemoryTuner chore. 2015-06-17 11:06:45,328 INFO [M:0;10.0.0.110:50903] regionserver.HRegionServer(1363): Serving as 10.0.0.110,50903,1434564404489, RpcServer on 10.0.0.110/10.0.0.110:50903, sessionid=0x14e02b3a2320000 2015-06-17 11:06:45,329 DEBUG [M:0;10.0.0.110:50903] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2015-06-17 11:06:45,329 DEBUG [M:0;10.0.0.110:50903] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager 10.0.0.110,50903,1434564404489 2015-06-17 11:06:45,329 DEBUG [M:0;10.0.0.110:50903] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,50903,1434564404489' 2015-06-17 11:06:45,330 DEBUG [M:0;10.0.0.110:50903] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2015-06-17 11:06:45,329 INFO [RS:0;10.0.0.110:50905] regionserver.HRegionServer(1363): Serving as 10.0.0.110,50905,1434564404764, RpcServer on 10.0.0.110/10.0.0.110:50905, sessionid=0x14e02b3a2320001 2015-06-17 11:06:45,330 DEBUG [RS:0;10.0.0.110:50905] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2015-06-17 11:06:45,330 DEBUG [RS:0;10.0.0.110:50905] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager 10.0.0.110,50905,1434564404764 2015-06-17 11:06:45,330 DEBUG [RS:0;10.0.0.110:50905] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,50905,1434564404764' 2015-06-17 11:06:45,331 DEBUG [RS:0;10.0.0.110:50905] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2015-06-17 11:06:45,331 DEBUG [M:0;10.0.0.110:50903] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2015-06-17 11:06:45,331 DEBUG [RS:0;10.0.0.110:50905] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2015-06-17 11:06:45,332 DEBUG [M:0;10.0.0.110:50903] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2015-06-17 11:06:45,332 DEBUG [M:0;10.0.0.110:50903] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2015-06-17 11:06:45,332 DEBUG [M:0;10.0.0.110:50903] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager 10.0.0.110,50903,1434564404489 2015-06-17 11:06:45,332 DEBUG [M:0;10.0.0.110:50903] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,50903,1434564404489' 2015-06-17 11:06:45,332 DEBUG [M:0;10.0.0.110:50903] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2015-06-17 11:06:45,332 DEBUG [RS:0;10.0.0.110:50905] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2015-06-17 11:06:45,332 DEBUG [RS:0;10.0.0.110:50905] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2015-06-17 11:06:45,332 DEBUG [RS:0;10.0.0.110:50905] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager 10.0.0.110,50905,1434564404764 2015-06-17 11:06:45,333 DEBUG [RS:0;10.0.0.110:50905] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,50905,1434564404764' 2015-06-17 11:06:45,333 DEBUG [RS:0;10.0.0.110:50905] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2015-06-17 11:06:45,334 DEBUG [M:0;10.0.0.110:50903] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2015-06-17 11:06:45,335 DEBUG [RS:0;10.0.0.110:50905] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2015-06-17 11:06:45,335 DEBUG [10.0.0.110:50903.activeMasterManager] zookeeper.ZKUtil(745): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2015-06-17 11:06:45,335 DEBUG [M:0;10.0.0.110:50903] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2015-06-17 11:06:45,335 INFO [10.0.0.110:50903.activeMasterManager] master.HMaster(884): Re-assigning hbase:meta with replicaId, 0 it was on null 2015-06-17 11:06:45,336 DEBUG [RS:0;10.0.0.110:50905] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2015-06-17 11:06:45,339 DEBUG [10.0.0.110:50903.activeMasterManager] master.AssignmentManager(1239): No previous transition plan found (or ignoring an existing plan) for hbase:meta,,1.1588230740; generated random plan=hri=hbase:meta,,1.1588230740, src=, dest=10.0.0.110,50903,1434564404489; 2 (online=2) available servers, forceNewPlan=false 2015-06-17 11:06:45,339 INFO [10.0.0.110:50903.activeMasterManager] master.AssignmentManager(1028): Assigning hbase:meta,,1.1588230740 to 10.0.0.110,50903,1434564404489 2015-06-17 11:06:45,340 INFO [10.0.0.110:50903.activeMasterManager] master.RegionStates(1072): Transition {1588230740 state=OFFLINE, ts=1434564405335, server=null} to {1588230740 state=PENDING_OPEN, ts=1434564405340, server=10.0.0.110,50903,1434564404489} 2015-06-17 11:06:45,340 INFO [10.0.0.110:50903.activeMasterManager] zookeeper.MetaTableLocator(440): Setting hbase:meta region location in ZooKeeper as 10.0.0.110,50903,1434564404489 2015-06-17 11:06:45,340 INFO [M:0;10.0.0.110:50903] quotas.RegionServerQuotaManager(62): Quota support disabled 2015-06-17 11:06:45,340 INFO [RS:0;10.0.0.110:50905] quotas.RegionServerQuotaManager(62): Quota support disabled 2015-06-17 11:06:45,348 DEBUG [10.0.0.110:50903.activeMasterManager] zookeeper.MetaTableLocator(452): META region location doesn't exist, create it 2015-06-17 11:06:45,350 DEBUG [10.0.0.110:50903.activeMasterManager] master.ServerManager(937): New admin connection to 10.0.0.110,50903,1434564404489 2015-06-17 11:06:45,370 INFO [10.0.0.110:50903.activeMasterManager] regionserver.RSRpcServices(1432): Open hbase:meta,,1.1588230740 2015-06-17 11:06:45,374 INFO [RS_OPEN_META-10.0.0.110:50903-0] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:06:45,379 INFO [RS_OPEN_META-10.0.0.110:50903-0] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.0.0.110%2C50903%2C1434564404489..meta, suffix=.meta, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50903,1434564404489, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:06:45,390 INFO [RS_OPEN_META-10.0.0.110:50903-0] wal.FSHLog(1365): Slow sync cost: 6 ms, current pipeline: [] 2015-06-17 11:06:45,390 INFO [RS_OPEN_META-10.0.0.110:50903-0] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50903,1434564404489/10.0.0.110%2C50903%2C1434564404489..meta.1434564405379.meta 2015-06-17 11:06:45,392 DEBUG [RS_OPEN_META-10.0.0.110:50903-0] regionserver.HRegion(6070): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2015-06-17 11:06:45,428 DEBUG [RS_OPEN_META-10.0.0.110:50903-0] coprocessor.CoprocessorHost(179): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2015-06-17 11:06:45,431 DEBUG [RS_OPEN_META-10.0.0.110:50903-0] regionserver.HRegion(7284): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2015-06-17 11:06:45,435 INFO [RS_OPEN_META-10.0.0.110:50903-0] regionserver.RegionCoprocessorHost(350): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2015-06-17 11:06:45,438 DEBUG [10.0.0.110:50903.activeMasterManager] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564405380,"tag":[],"qualifier":"state","vlen":2}]},"row":"hbase:meta"} 2015-06-17 11:06:45,440 DEBUG [RS_OPEN_META-10.0.0.110:50903-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table meta 1588230740 2015-06-17 11:06:45,440 DEBUG [RS_OPEN_META-10.0.0.110:50903-0] regionserver.HRegion(722): Instantiated hbase:meta,,1.1588230740 2015-06-17 11:06:45,448 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:06:45,448 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:06:45,450 DEBUG [StoreOpener-1588230740-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info 2015-06-17 11:06:45,451 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:06:45,452 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:06:45,453 DEBUG [StoreOpener-1588230740-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table 2015-06-17 11:06:45,457 DEBUG [RS_OPEN_META-10.0.0.110:50903-0] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740 2015-06-17 11:06:45,459 DEBUG [RS_OPEN_META-10.0.0.110:50903-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:06:45,465 DEBUG [RS_OPEN_META-10.0.0.110:50903-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/recovered.edits/3.seqid to file, newSeqId=3, maxSeqId=2 2015-06-17 11:06:45,466 INFO [RS_OPEN_META-10.0.0.110:50903-0] regionserver.HRegion(848): Onlined 1588230740; next sequenceid=3 2015-06-17 11:06:45,484 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1893): Post open deploy tasks for hbase:meta,,1.1588230740 2015-06-17 11:06:45,488 DEBUG [PostOpenDeployTasks:1588230740] master.AssignmentManager(2812): Got transition OPENED for {1588230740 state=PENDING_OPEN, ts=1434564405340, server=10.0.0.110,50903,1434564404489} from 10.0.0.110,50903,1434564404489 2015-06-17 11:06:45,488 INFO [PostOpenDeployTasks:1588230740] master.RegionStates(1072): Transition {1588230740 state=PENDING_OPEN, ts=1434564405340, server=10.0.0.110,50903,1434564404489} to {1588230740 state=OPEN, ts=1434564405488, server=10.0.0.110,50903,1434564404489} 2015-06-17 11:06:45,489 INFO [PostOpenDeployTasks:1588230740] zookeeper.MetaTableLocator(440): Setting hbase:meta region location in ZooKeeper as 10.0.0.110,50903,1434564404489 2015-06-17 11:06:45,491 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2015-06-17 11:06:45,491 DEBUG [PostOpenDeployTasks:1588230740] master.RegionStates(417): Onlined 1588230740 on 10.0.0.110,50903,1434564404489 2015-06-17 11:06:45,493 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1920): Finished post open deploy task for hbase:meta,,1.1588230740 2015-06-17 11:06:45,493 DEBUG [RS_OPEN_META-10.0.0.110:50903-0] handler.OpenRegionHandler(125): Opened hbase:meta,,1.1588230740 on 10.0.0.110,50903,1434564404489 2015-06-17 11:06:45,717 DEBUG [AsyncRpcChannel-pool2-t2] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service ClientService, sasl=false 2015-06-17 11:06:45,717 DEBUG [RpcServer.listener,port=50903] ipc.RpcServer$Listener(840): RpcServer.listener,port=50903: connection from 10.0.0.110:50923; # active connections: 2 2015-06-17 11:06:45,718 INFO [RpcServer.reader=2,bindAddress=10.0.0.110,port=50903] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 50923 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:06:45,746 INFO [10.0.0.110:50903.activeMasterManager] hbase.MetaTableAccessor(1692): Updated table hbase:meta state to ENABLED in META 2015-06-17 11:06:45,746 DEBUG [10.0.0.110:50903.activeMasterManager] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564405746,"tag":[],"qualifier":"state","vlen":2}]},"row":"hbase:meta"} 2015-06-17 11:06:45,751 INFO [10.0.0.110:50903.activeMasterManager] hbase.MetaTableAccessor(1692): Updated table hbase:meta state to ENABLED in META 2015-06-17 11:06:45,806 INFO [10.0.0.110:50903.activeMasterManager] master.ServerManager(686): AssignmentManager hasn't finished failover cleanup; waiting 2015-06-17 11:06:45,807 INFO [10.0.0.110:50903.activeMasterManager] master.HMaster(911): hbase:meta with replicaId 0 assigned=1, location=10.0.0.110,50903,1434564404489 2015-06-17 11:06:45,829 INFO [10.0.0.110:50903.activeMasterManager] master.AssignmentManager(538): Clean cluster startup. Assigning user regions 2015-06-17 11:06:45,840 INFO [10.0.0.110:50903.activeMasterManager] master.AssignmentManager(420): Joined the cluster in 30ms, failover=false 2015-06-17 11:06:45,851 INFO [10.0.0.110:50903.activeMasterManager] master.TableNamespaceManager(90): Namespace table not found. Creating... 2015-06-17 11:06:45,997 DEBUG [10.0.0.110:50903.activeMasterManager] procedure2.ProcedureExecutor(566): Procedure CreateTableProcedure (table=hbase:namespace) user=ndimiduk (auth:SIMPLE) id=1 state=RUNNABLE added to the store. 2015-06-17 11:06:46,015 DEBUG [ProcedureExecutorThread-0] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/hbase:namespace/write-master:509030000000000 2015-06-17 11:06:46,140 INFO [IPC Server handler 5 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741833_1009{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 0 2015-06-17 11:06:46,143 DEBUG [ProcedureExecutorThread-0] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2015-06-17 11:06:46,147 INFO [RegionOpenAndInitThread-hbase:namespace-1] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:06:46,152 INFO [RegionOpenAndInitThread-hbase:namespace-1] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=hregion-72409285.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-72409285, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/oldWALs 2015-06-17 11:06:46,159 INFO [RegionOpenAndInitThread-hbase:namespace-1] wal.FSHLog(1365): Slow sync cost: 4 ms, current pipeline: [] 2015-06-17 11:06:46,159 INFO [RegionOpenAndInitThread-hbase:namespace-1] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-72409285/hregion-72409285.default.1434564406152 2015-06-17 11:06:46,160 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(5888): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '10', TTL => 'FOREVER', MIN_VERSIONS => '0', CACHE_DATA_IN_L1 => 'true', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '8192', IN_MEMORY => 'true', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp Table name == hbase:namespace 2015-06-17 11:06:46,169 INFO [IPC Server handler 5 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741835_1011{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 0 2015-06-17 11:06:46,170 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(722): Instantiated hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:06:46,170 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1365): Closing hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.: disabling compactions & flushes 2015-06-17 11:06:46,170 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1392): Updates disabled for region hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:06:46,170 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1486): Closed hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:06:46,171 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-72409285 2015-06-17 11:06:46,173 INFO [IPC Server handler 7 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741834_1010{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 83 2015-06-17 11:06:46,178 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/oldWALs 2015-06-17 11:06:46,178 INFO [RegionOpenAndInitThread-hbase:namespace-1] wal.FSHLog(977): Closed WAL: FSHLog hregion-72409285.default:(num 1434564406152) 2015-06-17 11:06:46,291 DEBUG [ProcedureExecutorThread-0] hbase.MetaTableAccessor(1371): Put{"totalColumns":1,"families":{"info":[{"timestamp":9223372036854775807,"tag":[],"qualifier":"regioninfo","vlen":41}]},"row":"hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0."} 2015-06-17 11:06:46,295 INFO [ProcedureExecutorThread-0] hbase.MetaTableAccessor(1568): Added 1 2015-06-17 11:06:46,401 INFO [ProcedureExecutorThread-0] master.AssignmentManager(680): Assigning 1 region(s) to 10.0.0.110,50903,1434564404489 2015-06-17 11:06:46,403 ERROR [ProcedureExecutorThread-0] master.TableStateManager(133): Unable to get table hbase:namespace state, probably table not exists 2015-06-17 11:06:46,404 INFO [ProcedureExecutorThread-0] master.RegionStates(1072): Transition {9ccc7b1e8942a5472c9cf2d844c2fff0 state=OFFLINE, ts=1434564406400, server=null} to {9ccc7b1e8942a5472c9cf2d844c2fff0 state=PENDING_OPEN, ts=1434564406404, server=10.0.0.110,50903,1434564404489} 2015-06-17 11:06:46,404 INFO [ProcedureExecutorThread-0] master.RegionStateStore(207): Updating row hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. with state=PENDING_OPEN&sn=10.0.0.110,50903,1434564404489 2015-06-17 11:06:46,406 INFO [ProcedureExecutorThread-0] regionserver.RSRpcServices(1432): Open hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:06:46,411 DEBUG [ProcedureExecutorThread-0] master.AssignmentManager(851): Bulk assigning done for 10.0.0.110,50903,1434564404489 2015-06-17 11:06:46,412 DEBUG [RS_OPEN_REGION-10.0.0.110:50903-0] regionserver.HRegion(6070): Opening region: {ENCODED => 9ccc7b1e8942a5472c9cf2d844c2fff0, NAME => 'hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.', STARTKEY => '', ENDKEY => ''} 2015-06-17 11:06:46,412 DEBUG [ProcedureExecutorThread-0] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564406412,"tag":[],"qualifier":"state","vlen":2}]},"row":"hbase:namespace"} 2015-06-17 11:06:46,412 DEBUG [RS_OPEN_REGION-10.0.0.110:50903-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table namespace 9ccc7b1e8942a5472c9cf2d844c2fff0 2015-06-17 11:06:46,412 DEBUG [RS_OPEN_REGION-10.0.0.110:50903-0] regionserver.HRegion(722): Instantiated hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:06:46,415 INFO [ProcedureExecutorThread-0] hbase.MetaTableAccessor(1692): Updated table hbase:namespace state to ENABLED in META 2015-06-17 11:06:46,417 INFO [StoreOpener-9ccc7b1e8942a5472c9cf2d844c2fff0-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:06:46,417 INFO [StoreOpener-9ccc7b1e8942a5472c9cf2d844c2fff0-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:06:46,419 DEBUG [StoreOpener-9ccc7b1e8942a5472c9cf2d844c2fff0-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/info 2015-06-17 11:06:46,420 DEBUG [RS_OPEN_REGION-10.0.0.110:50903-0] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0 2015-06-17 11:06:46,423 DEBUG [RS_OPEN_REGION-10.0.0.110:50903-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:06:46,428 DEBUG [RS_OPEN_REGION-10.0.0.110:50903-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-06-17 11:06:46,428 INFO [RS_OPEN_REGION-10.0.0.110:50903-0] regionserver.HRegion(848): Onlined 9ccc7b1e8942a5472c9cf2d844c2fff0; next sequenceid=2 2015-06-17 11:06:46,430 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] regionserver.HRegionServer(1893): Post open deploy tasks for hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:06:46,430 DEBUG [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.AssignmentManager(2812): Got transition OPENED for {9ccc7b1e8942a5472c9cf2d844c2fff0 state=PENDING_OPEN, ts=1434564406404, server=10.0.0.110,50903,1434564404489} from 10.0.0.110,50903,1434564404489 2015-06-17 11:06:46,430 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStates(1072): Transition {9ccc7b1e8942a5472c9cf2d844c2fff0 state=PENDING_OPEN, ts=1434564406404, server=10.0.0.110,50903,1434564404489} to {9ccc7b1e8942a5472c9cf2d844c2fff0 state=OPEN, ts=1434564406430, server=10.0.0.110,50903,1434564404489} 2015-06-17 11:06:46,431 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStateStore(207): Updating row hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. with state=OPEN&openSeqNum=2&server=10.0.0.110,50903,1434564404489 2015-06-17 11:06:46,432 DEBUG [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStates(417): Onlined 9ccc7b1e8942a5472c9cf2d844c2fff0 on 10.0.0.110,50903,1434564404489 2015-06-17 11:06:46,438 DEBUG [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] regionserver.HRegionServer(1920): Finished post open deploy task for hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:06:46,439 DEBUG [RS_OPEN_REGION-10.0.0.110:50903-0] handler.OpenRegionHandler(125): Opened hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. on 10.0.0.110,50903,1434564404489 2015-06-17 11:06:46,509 DEBUG [10.0.0.110:50903.activeMasterManager] zookeeper.ZKUtil(488): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2015-06-17 11:06:46,510 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2015-06-17 11:06:46,535 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2015-06-17 11:06:46,539 DEBUG [main-EventThread] hbase.ZKNamespaceManager(196): Updating namespace cache from node default with data: \x0A\x07default 2015-06-17 11:06:46,544 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2015-06-17 11:06:46,546 DEBUG [main-EventThread] hbase.ZKNamespaceManager(196): Updating namespace cache from node default with data: \x0A\x07default 2015-06-17 11:06:46,546 DEBUG [main-EventThread] hbase.ZKNamespaceManager(196): Updating namespace cache from node hbase with data: \x0A\x05hbase 2015-06-17 11:06:46,550 DEBUG [10.0.0.110:50903.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/namespace/default already exists 2015-06-17 11:06:46,551 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2015-06-17 11:06:46,551 DEBUG [10.0.0.110:50903.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/namespace/hbase already exists 2015-06-17 11:06:46,552 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2015-06-17 11:06:46,552 INFO [10.0.0.110:50903.activeMasterManager] master.HMaster(760): Master has completed initialization 2015-06-17 11:06:46,555 INFO [10.0.0.110:50903.activeMasterManager] quotas.MasterQuotaManager(71): Quota support disabled 2015-06-17 11:06:46,639 INFO [main] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x51e5f22a connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:06:46,642 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x51e5f22a0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:06:46,643 DEBUG [main] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f673724, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:06:46,643 DEBUG [main] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:06:46,643 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x51e5f22a-0x14e02b3a2320008 connected 2015-06-17 11:06:46,643 DEBUG [main] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:06:46,646 DEBUG [AsyncRpcChannel-pool2-t3] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service ClientService, sasl=false 2015-06-17 11:06:46,646 DEBUG [RpcServer.listener,port=50903] ipc.RpcServer$Listener(840): RpcServer.listener,port=50903: connection from 10.0.0.110:50929; # active connections: 3 2015-06-17 11:06:46,647 INFO [RpcServer.reader=3,bindAddress=10.0.0.110,port=50903] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 50929 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:06:46,655 INFO [main] hbase.HBaseTestingUtility(1037): Minicluster is up 2015-06-17 11:06:46,656 INFO [main] hbase.HBaseTestingUtility(1211): The hbase.fs.tmp.dir is set to /user/ndimiduk/hbase-staging 2015-06-17 11:06:46,657 DEBUG [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(188): Sleeping for 6273 before executing next master restart count=0 2015-06-17 11:06:46,657 INFO [Thread-199] procedure.TestMasterRestartWithProcedures$1(104): executing create table step=0 2015-06-17 11:06:46,666 DEBUG [AsyncRpcChannel-pool2-t4] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service MasterService, sasl=false 2015-06-17 11:06:46,667 DEBUG [RpcServer.listener,port=50903] ipc.RpcServer$Listener(840): RpcServer.listener,port=50903: connection from 10.0.0.110:50930; # active connections: 4 2015-06-17 11:06:46,668 INFO [RpcServer.reader=4,bindAddress=10.0.0.110,port=50903] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 50930 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:06:46,677 INFO [B.defaultRpcServer.handler=1,queue=0,port=50903] master.HMaster(1361): Client=ndimiduk/null create 'testMasterProcsWithMasterRestarting', {NAME => 'f1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2015-06-17 11:06:46,725 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50903] procedure2.ProcedureExecutor(566): Procedure CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=2 state=RUNNABLE added to the store. 2015-06-17 11:06:46,727 DEBUG [ProcedureExecutorThread-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/hbase:namespace/write-master:509030000000000 2015-06-17 11:06:46,728 DEBUG [ProcedureExecutorThread-0] procedure2.ProcedureExecutor(729): Procedure completed in 768msec: CreateTableProcedure (table=hbase:namespace) user=ndimiduk (auth:SIMPLE) id=1 state=FINISHED 2015-06-17 11:06:46,732 DEBUG [ProcedureExecutorThread-1] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testMasterProcsWithMasterRestarting/write-master:509030000000000 2015-06-17 11:06:46,738 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=2 2015-06-17 11:06:46,742 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=2 2015-06-17 11:06:46,746 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=2 2015-06-17 11:06:46,751 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=2 2015-06-17 11:06:46,759 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=2 2015-06-17 11:06:46,773 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=2 2015-06-17 11:06:46,797 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=2 2015-06-17 11:06:46,840 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=2 2015-06-17 11:06:46,849 INFO [IPC Server handler 7 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741836_1012{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 0 2015-06-17 11:06:46,852 DEBUG [ProcedureExecutorThread-1] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/.tabledesc/.tableinfo.0000000001 2015-06-17 11:06:46,853 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:06:46,853 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:06:46,853 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:06:46,853 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:06:46,857 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=hregion-47207677.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-47207677, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/oldWALs 2015-06-17 11:06:46,858 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=hregion-10563189.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-10563189, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/oldWALs 2015-06-17 11:06:46,858 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=hregion-01114407.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-01114407, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/oldWALs 2015-06-17 11:06:46,858 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=hregion-69315643.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-69315643, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/oldWALs 2015-06-17 11:06:46,867 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] wal.FSHLog(1365): Slow sync cost: 6 ms, current pipeline: [] 2015-06-17 11:06:46,867 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] wal.FSHLog(1365): Slow sync cost: 6 ms, current pipeline: [] 2015-06-17 11:06:46,868 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-10563189/hregion-10563189.default.1434564406858 2015-06-17 11:06:46,868 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-47207677/hregion-47207677.default.1434564406858 2015-06-17 11:06:46,868 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] wal.FSHLog(1365): Slow sync cost: 6 ms, current pipeline: [] 2015-06-17 11:06:46,868 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-01114407/hregion-01114407.default.1434564406858 2015-06-17 11:06:46,869 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] regionserver.HRegion(5888): creating HRegion testMasterProcsWithMasterRestarting HTD == 'testMasterProcsWithMasterRestarting', {NAME => 'f1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp Table name == testMasterProcsWithMasterRestarting 2015-06-17 11:06:46,869 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] regionserver.HRegion(5888): creating HRegion testMasterProcsWithMasterRestarting HTD == 'testMasterProcsWithMasterRestarting', {NAME => 'f1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp Table name == testMasterProcsWithMasterRestarting 2015-06-17 11:06:46,869 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] wal.FSHLog(1365): Slow sync cost: 7 ms, current pipeline: [] 2015-06-17 11:06:46,870 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-69315643/hregion-69315643.default.1434564406858 2015-06-17 11:06:46,870 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] regionserver.HRegion(5888): creating HRegion testMasterProcsWithMasterRestarting HTD == 'testMasterProcsWithMasterRestarting', {NAME => 'f1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp Table name == testMasterProcsWithMasterRestarting 2015-06-17 11:06:46,871 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] regionserver.HRegion(5888): creating HRegion testMasterProcsWithMasterRestarting HTD == 'testMasterProcsWithMasterRestarting', {NAME => 'f1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp Table name == testMasterProcsWithMasterRestarting 2015-06-17 11:06:46,885 INFO [IPC Server handler 4 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741841_1017{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 0 2015-06-17 11:06:46,887 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] regionserver.HRegion(722): Instantiated testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. 2015-06-17 11:06:46,888 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] regionserver.HRegion(1365): Closing testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de.: disabling compactions & flushes 2015-06-17 11:06:46,888 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] regionserver.HRegion(1392): Updates disabled for region testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. 2015-06-17 11:06:46,890 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] regionserver.HRegion(1486): Closed testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. 2015-06-17 11:06:46,890 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-10563189 2015-06-17 11:06:46,890 INFO [IPC Server handler 7 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741842_1018{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 0 2015-06-17 11:06:46,893 INFO [IPC Server handler 8 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741844_1020{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 0 2015-06-17 11:06:46,893 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] regionserver.HRegion(722): Instantiated testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. 2015-06-17 11:06:46,893 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] regionserver.HRegion(1365): Closing testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626.: disabling compactions & flushes 2015-06-17 11:06:46,893 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] regionserver.HRegion(1392): Updates disabled for region testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. 2015-06-17 11:06:46,893 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] regionserver.HRegion(1486): Closed testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. 2015-06-17 11:06:46,893 INFO [IPC Server handler 8 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741843_1019{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 72 2015-06-17 11:06:46,893 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-47207677 2015-06-17 11:06:46,895 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] regionserver.HRegion(722): Instantiated testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. 2015-06-17 11:06:46,895 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] regionserver.HRegion(1365): Closing testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb.: disabling compactions & flushes 2015-06-17 11:06:46,895 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] regionserver.HRegion(1392): Updates disabled for region testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. 2015-06-17 11:06:46,896 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] regionserver.HRegion(1486): Closed testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. 2015-06-17 11:06:46,896 INFO [IPC Server handler 1 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741838_1014{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 91 2015-06-17 11:06:46,896 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-69315643 2015-06-17 11:06:46,898 INFO [IPC Server handler 2 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741837_1013{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 91 2015-06-17 11:06:46,899 INFO [IPC Server handler 7 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741840_1016{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 91 2015-06-17 11:06:46,943 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=2 2015-06-17 11:06:47,047 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=2 2015-06-17 11:06:47,149 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=2 2015-06-17 11:06:47,252 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=2 2015-06-17 11:06:47,295 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] regionserver.HRegion(722): Instantiated testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. 2015-06-17 11:06:47,296 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] regionserver.HRegion(1365): Closing testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405.: disabling compactions & flushes 2015-06-17 11:06:47,296 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] regionserver.HRegion(1392): Updates disabled for region testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. 2015-06-17 11:06:47,296 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] regionserver.HRegion(1486): Closed testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. 2015-06-17 11:06:47,296 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-01114407 2015-06-17 11:06:47,300 INFO [IPC Server handler 3 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741839_1015{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 83 2015-06-17 11:06:47,303 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/oldWALs 2015-06-17 11:06:47,303 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] wal.FSHLog(977): Closed WAL: FSHLog hregion-10563189.default:(num 1434564406858) 2015-06-17 11:06:47,304 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/oldWALs 2015-06-17 11:06:47,304 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] wal.FSHLog(977): Closed WAL: FSHLog hregion-47207677.default:(num 1434564406858) 2015-06-17 11:06:47,305 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/oldWALs 2015-06-17 11:06:47,305 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] wal.FSHLog(977): Closed WAL: FSHLog hregion-69315643.default:(num 1434564406858) 2015-06-17 11:06:47,305 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/oldWALs 2015-06-17 11:06:47,305 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] wal.FSHLog(977): Closed WAL: FSHLog hregion-01114407.default:(num 1434564406858) 2015-06-17 11:06:47,417 DEBUG [ProcedureExecutorThread-1] hbase.MetaTableAccessor(1371): Put{"totalColumns":1,"families":{"info":[{"timestamp":9223372036854775807,"tag":[],"qualifier":"regioninfo","vlen":70}]},"row":"testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de."}, Put{"totalColumns":1,"families":{"info":[{"timestamp":9223372036854775807,"tag":[],"qualifier":"regioninfo","vlen":71}]},"row":"testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626."}, Put{"totalColumns":1,"families":{"info":[{"timestamp":9223372036854775807,"tag":[],"qualifier":"regioninfo","vlen":70}]},"row":"testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb."}, Put{"totalColumns":1,"families":{"info":[{"timestamp":9223372036854775807,"tag":[],"qualifier":"regioninfo","vlen":71}]},"row":"testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405."} 2015-06-17 11:06:47,422 INFO [ProcedureExecutorThread-1] hbase.MetaTableAccessor(1568): Added 4 2015-06-17 11:06:47,456 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=2 2015-06-17 11:06:47,526 INFO [ProcedureExecutorThread-1] master.AssignmentManager(680): Assigning 4 region(s) to 10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,528 ERROR [ProcedureExecutorThread-1] master.TableStateManager(133): Unable to get table testMasterProcsWithMasterRestarting state, probably table not exists 2015-06-17 11:06:47,530 ERROR [ProcedureExecutorThread-1] master.TableStateManager(133): Unable to get table testMasterProcsWithMasterRestarting state, probably table not exists 2015-06-17 11:06:47,531 ERROR [ProcedureExecutorThread-1] master.TableStateManager(133): Unable to get table testMasterProcsWithMasterRestarting state, probably table not exists 2015-06-17 11:06:47,534 ERROR [ProcedureExecutorThread-1] master.TableStateManager(133): Unable to get table testMasterProcsWithMasterRestarting state, probably table not exists 2015-06-17 11:06:47,534 INFO [ProcedureExecutorThread-1] master.RegionStates(1072): Transition {c555b4ca922c2de1fdd394fcc2ac20de state=OFFLINE, ts=1434564407526, server=null} to {c555b4ca922c2de1fdd394fcc2ac20de state=PENDING_OPEN, ts=1434564407534, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:47,534 INFO [ProcedureExecutorThread-1] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. with state=PENDING_OPEN&sn=10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,536 INFO [ProcedureExecutorThread-1] master.RegionStates(1072): Transition {3216ee6f55befe8693a09a4e6457d626 state=OFFLINE, ts=1434564407526, server=null} to {3216ee6f55befe8693a09a4e6457d626 state=PENDING_OPEN, ts=1434564407536, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:47,536 INFO [ProcedureExecutorThread-1] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. with state=PENDING_OPEN&sn=10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,537 INFO [ProcedureExecutorThread-1] master.RegionStates(1072): Transition {3819afeab7512f52d645bda087c8d2eb state=OFFLINE, ts=1434564407526, server=null} to {3819afeab7512f52d645bda087c8d2eb state=PENDING_OPEN, ts=1434564407537, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:47,537 INFO [ProcedureExecutorThread-1] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. with state=PENDING_OPEN&sn=10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,538 INFO [ProcedureExecutorThread-1] master.RegionStates(1072): Transition {7855514cb57ce135029b4fee0d052405 state=OFFLINE, ts=1434564407526, server=null} to {7855514cb57ce135029b4fee0d052405 state=PENDING_OPEN, ts=1434564407538, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:47,538 INFO [ProcedureExecutorThread-1] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. with state=PENDING_OPEN&sn=10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,540 DEBUG [ProcedureExecutorThread-1] master.ServerManager(937): New admin connection to 10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,543 DEBUG [AsyncRpcChannel-pool2-t5] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service AdminService, sasl=false 2015-06-17 11:06:47,543 DEBUG [RpcServer.listener,port=50905] ipc.RpcServer$Listener(840): RpcServer.listener,port=50905: connection from 10.0.0.110:50940; # active connections: 1 2015-06-17 11:06:47,544 INFO [RpcServer.reader=1,bindAddress=10.0.0.110,port=50905] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 50940 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:06:47,544 INFO [PriorityRpcServer.handler=0,queue=0,port=50905] regionserver.RSRpcServices(1432): Open testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. 2015-06-17 11:06:47,550 INFO [PriorityRpcServer.handler=0,queue=0,port=50905] regionserver.RSRpcServices(1432): Open testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. 2015-06-17 11:06:47,550 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.HRegion(6070): Opening region: {ENCODED => c555b4ca922c2de1fdd394fcc2ac20de, NAME => 'testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de.', STARTKEY => '', ENDKEY => 'a'} 2015-06-17 11:06:47,550 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testMasterProcsWithMasterRestarting c555b4ca922c2de1fdd394fcc2ac20de 2015-06-17 11:06:47,550 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.HRegion(722): Instantiated testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. 2015-06-17 11:06:47,551 INFO [PriorityRpcServer.handler=0,queue=0,port=50905] regionserver.RSRpcServices(1432): Open testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. 2015-06-17 11:06:47,551 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-1] regionserver.HRegion(6070): Opening region: {ENCODED => 3216ee6f55befe8693a09a4e6457d626, NAME => 'testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626.', STARTKEY => 'a', ENDKEY => 'b'} 2015-06-17 11:06:47,551 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testMasterProcsWithMasterRestarting 3216ee6f55befe8693a09a4e6457d626 2015-06-17 11:06:47,552 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-1] regionserver.HRegion(722): Instantiated testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. 2015-06-17 11:06:47,552 INFO [PriorityRpcServer.handler=0,queue=0,port=50905] regionserver.RSRpcServices(1432): Open testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. 2015-06-17 11:06:47,552 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-2] regionserver.HRegion(6070): Opening region: {ENCODED => 3819afeab7512f52d645bda087c8d2eb, NAME => 'testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb.', STARTKEY => 'c', ENDKEY => ''} 2015-06-17 11:06:47,553 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-2] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testMasterProcsWithMasterRestarting 3819afeab7512f52d645bda087c8d2eb 2015-06-17 11:06:47,553 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-2] regionserver.HRegion(722): Instantiated testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. 2015-06-17 11:06:47,555 INFO [StoreOpener-c555b4ca922c2de1fdd394fcc2ac20de-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:06:47,555 INFO [StoreOpener-3216ee6f55befe8693a09a4e6457d626-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:06:47,556 INFO [StoreOpener-c555b4ca922c2de1fdd394fcc2ac20de-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:06:47,556 INFO [StoreOpener-3216ee6f55befe8693a09a4e6457d626-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:06:47,557 INFO [StoreOpener-3819afeab7512f52d645bda087c8d2eb-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:06:47,557 DEBUG [StoreOpener-c555b4ca922c2de1fdd394fcc2ac20de-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/c555b4ca922c2de1fdd394fcc2ac20de/f1 2015-06-17 11:06:47,557 DEBUG [StoreOpener-3216ee6f55befe8693a09a4e6457d626-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/3216ee6f55befe8693a09a4e6457d626/f1 2015-06-17 11:06:47,557 INFO [StoreOpener-3819afeab7512f52d645bda087c8d2eb-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:06:47,558 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/c555b4ca922c2de1fdd394fcc2ac20de 2015-06-17 11:06:47,558 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-1] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/3216ee6f55befe8693a09a4e6457d626 2015-06-17 11:06:47,559 DEBUG [StoreOpener-3819afeab7512f52d645bda087c8d2eb-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/3819afeab7512f52d645bda087c8d2eb/f1 2015-06-17 11:06:47,560 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-2] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/3819afeab7512f52d645bda087c8d2eb 2015-06-17 11:06:47,561 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:06:47,561 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-1] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:06:47,562 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-2] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:06:47,567 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/c555b4ca922c2de1fdd394fcc2ac20de/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-06-17 11:06:47,567 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/3216ee6f55befe8693a09a4e6457d626/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-06-17 11:06:47,567 INFO [RS_OPEN_REGION-10.0.0.110:50905-1] regionserver.HRegion(848): Onlined 3216ee6f55befe8693a09a4e6457d626; next sequenceid=2 2015-06-17 11:06:47,567 INFO [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.HRegion(848): Onlined c555b4ca922c2de1fdd394fcc2ac20de; next sequenceid=2 2015-06-17 11:06:47,568 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-2] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/3819afeab7512f52d645bda087c8d2eb/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-06-17 11:06:47,568 INFO [RS_OPEN_REGION-10.0.0.110:50905-2] regionserver.HRegion(848): Onlined 3819afeab7512f52d645bda087c8d2eb; next sequenceid=2 2015-06-17 11:06:47,568 INFO [PostOpenDeployTasks:3216ee6f55befe8693a09a4e6457d626] regionserver.HRegionServer(1893): Post open deploy tasks for testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. 2015-06-17 11:06:47,568 INFO [PostOpenDeployTasks:c555b4ca922c2de1fdd394fcc2ac20de] regionserver.HRegionServer(1893): Post open deploy tasks for testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. 2015-06-17 11:06:47,569 INFO [PostOpenDeployTasks:3819afeab7512f52d645bda087c8d2eb] regionserver.HRegionServer(1893): Post open deploy tasks for testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. 2015-06-17 11:06:47,570 DEBUG [PriorityRpcServer.handler=15,queue=1,port=50903] master.AssignmentManager(2812): Got transition OPENED for {c555b4ca922c2de1fdd394fcc2ac20de state=PENDING_OPEN, ts=1434564407534, server=10.0.0.110,50905,1434564404764} from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,570 DEBUG [PriorityRpcServer.handler=16,queue=0,port=50903] master.AssignmentManager(2812): Got transition OPENED for {3216ee6f55befe8693a09a4e6457d626 state=PENDING_OPEN, ts=1434564407536, server=10.0.0.110,50905,1434564404764} from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,570 DEBUG [PriorityRpcServer.handler=18,queue=0,port=50903] master.AssignmentManager(2812): Got transition OPENED for {3819afeab7512f52d645bda087c8d2eb state=PENDING_OPEN, ts=1434564407537, server=10.0.0.110,50905,1434564404764} from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,570 INFO [PriorityRpcServer.handler=15,queue=1,port=50903] master.RegionStates(1072): Transition {c555b4ca922c2de1fdd394fcc2ac20de state=PENDING_OPEN, ts=1434564407534, server=10.0.0.110,50905,1434564404764} to {c555b4ca922c2de1fdd394fcc2ac20de state=OPEN, ts=1434564407570, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:47,570 INFO [PriorityRpcServer.handler=18,queue=0,port=50903] master.RegionStates(1072): Transition {3819afeab7512f52d645bda087c8d2eb state=PENDING_OPEN, ts=1434564407537, server=10.0.0.110,50905,1434564404764} to {3819afeab7512f52d645bda087c8d2eb state=OPEN, ts=1434564407570, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:47,570 INFO [PriorityRpcServer.handler=16,queue=0,port=50903] master.RegionStates(1072): Transition {3216ee6f55befe8693a09a4e6457d626 state=PENDING_OPEN, ts=1434564407536, server=10.0.0.110,50905,1434564404764} to {3216ee6f55befe8693a09a4e6457d626 state=OPEN, ts=1434564407570, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:47,570 INFO [PriorityRpcServer.handler=15,queue=1,port=50903] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. with state=OPEN&openSeqNum=2&server=10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,570 INFO [PriorityRpcServer.handler=16,queue=0,port=50903] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. with state=OPEN&openSeqNum=2&server=10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,570 INFO [PriorityRpcServer.handler=18,queue=0,port=50903] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. with state=OPEN&openSeqNum=2&server=10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,572 DEBUG [PriorityRpcServer.handler=15,queue=1,port=50903] master.RegionStates(417): Onlined c555b4ca922c2de1fdd394fcc2ac20de on 10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,572 DEBUG [PriorityRpcServer.handler=16,queue=0,port=50903] master.RegionStates(417): Onlined 3216ee6f55befe8693a09a4e6457d626 on 10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,573 DEBUG [PriorityRpcServer.handler=18,queue=0,port=50903] master.RegionStates(417): Onlined 3819afeab7512f52d645bda087c8d2eb on 10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,574 ERROR [PriorityRpcServer.handler=15,queue=1,port=50903] master.TableStateManager(133): Unable to get table testMasterProcsWithMasterRestarting state, probably table not exists 2015-06-17 11:06:47,574 DEBUG [PostOpenDeployTasks:c555b4ca922c2de1fdd394fcc2ac20de] regionserver.HRegionServer(1920): Finished post open deploy task for testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. 2015-06-17 11:06:47,574 ERROR [PriorityRpcServer.handler=16,queue=0,port=50903] master.TableStateManager(133): Unable to get table testMasterProcsWithMasterRestarting state, probably table not exists 2015-06-17 11:06:47,574 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] handler.OpenRegionHandler(125): Opened testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. on 10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,575 ERROR [PriorityRpcServer.handler=18,queue=0,port=50903] master.TableStateManager(133): Unable to get table testMasterProcsWithMasterRestarting state, probably table not exists 2015-06-17 11:06:47,575 DEBUG [PostOpenDeployTasks:3216ee6f55befe8693a09a4e6457d626] regionserver.HRegionServer(1920): Finished post open deploy task for testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. 2015-06-17 11:06:47,575 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-1] handler.OpenRegionHandler(125): Opened testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. on 10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,575 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.HRegion(6070): Opening region: {ENCODED => 7855514cb57ce135029b4fee0d052405, NAME => 'testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405.', STARTKEY => 'b', ENDKEY => 'c'} 2015-06-17 11:06:47,575 DEBUG [PostOpenDeployTasks:3819afeab7512f52d645bda087c8d2eb] regionserver.HRegionServer(1920): Finished post open deploy task for testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. 2015-06-17 11:06:47,575 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-2] handler.OpenRegionHandler(125): Opened testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. on 10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,576 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testMasterProcsWithMasterRestarting 7855514cb57ce135029b4fee0d052405 2015-06-17 11:06:47,576 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.HRegion(722): Instantiated testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. 2015-06-17 11:06:47,579 INFO [StoreOpener-7855514cb57ce135029b4fee0d052405-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:06:47,579 INFO [StoreOpener-7855514cb57ce135029b4fee0d052405-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:06:47,580 DEBUG [StoreOpener-7855514cb57ce135029b4fee0d052405-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/7855514cb57ce135029b4fee0d052405/f1 2015-06-17 11:06:47,581 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/7855514cb57ce135029b4fee0d052405 2015-06-17 11:06:47,584 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:06:47,587 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/7855514cb57ce135029b4fee0d052405/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-06-17 11:06:47,587 INFO [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.HRegion(848): Onlined 7855514cb57ce135029b4fee0d052405; next sequenceid=2 2015-06-17 11:06:47,588 INFO [PostOpenDeployTasks:7855514cb57ce135029b4fee0d052405] regionserver.HRegionServer(1893): Post open deploy tasks for testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. 2015-06-17 11:06:47,589 DEBUG [PriorityRpcServer.handler=2,queue=0,port=50903] master.AssignmentManager(2812): Got transition OPENED for {7855514cb57ce135029b4fee0d052405 state=PENDING_OPEN, ts=1434564407538, server=10.0.0.110,50905,1434564404764} from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,589 INFO [PriorityRpcServer.handler=2,queue=0,port=50903] master.RegionStates(1072): Transition {7855514cb57ce135029b4fee0d052405 state=PENDING_OPEN, ts=1434564407538, server=10.0.0.110,50905,1434564404764} to {7855514cb57ce135029b4fee0d052405 state=OPEN, ts=1434564407589, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:47,589 INFO [PriorityRpcServer.handler=2,queue=0,port=50903] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. with state=OPEN&openSeqNum=2&server=10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,591 DEBUG [PriorityRpcServer.handler=2,queue=0,port=50903] master.RegionStates(417): Onlined 7855514cb57ce135029b4fee0d052405 on 10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,591 DEBUG [ProcedureExecutorThread-1] master.AssignmentManager(851): Bulk assigning done for 10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,591 DEBUG [ProcedureExecutorThread-1] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564407591,"tag":[],"qualifier":"state","vlen":2}]},"row":"testMasterProcsWithMasterRestarting"} 2015-06-17 11:06:47,592 ERROR [PriorityRpcServer.handler=2,queue=0,port=50903] master.TableStateManager(133): Unable to get table testMasterProcsWithMasterRestarting state, probably table not exists 2015-06-17 11:06:47,592 DEBUG [PostOpenDeployTasks:7855514cb57ce135029b4fee0d052405] regionserver.HRegionServer(1920): Finished post open deploy task for testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. 2015-06-17 11:06:47,593 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] handler.OpenRegionHandler(125): Opened testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. on 10.0.0.110,50905,1434564404764 2015-06-17 11:06:47,594 INFO [ProcedureExecutorThread-1] hbase.MetaTableAccessor(1692): Updated table testMasterProcsWithMasterRestarting state to ENABLED in META 2015-06-17 11:06:47,661 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=2 2015-06-17 11:06:47,865 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=2 2015-06-17 11:06:47,912 DEBUG [ProcedureExecutorThread-1] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testMasterProcsWithMasterRestarting/write-master:509030000000000 2015-06-17 11:06:47,912 DEBUG [ProcedureExecutorThread-1] procedure2.ProcedureExecutor(729): Procedure completed in 1.1290sec: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=2 state=FINISHED 2015-06-17 11:06:48,067 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=2 2015-06-17 11:06:48,068 INFO [Thread-199] client.HBaseAdmin$TableFuture(4327): Operation: CREATE, Table Name: default:testMasterProcsWithMasterRestarting completed 2015-06-17 11:06:48,070 INFO [Thread-199] procedure.TestMasterRestartWithProcedures$2(119): executing disable table step=0 2015-06-17 11:06:48,072 INFO [Thread-199] client.HBaseAdmin$9(1131): Started disable of testMasterProcsWithMasterRestarting 2015-06-17 11:06:48,077 INFO [B.defaultRpcServer.handler=0,queue=0,port=50903] master.HMaster(1743): Client=ndimiduk/null disable testMasterProcsWithMasterRestarting 2015-06-17 11:06:48,187 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50903] procedure2.ProcedureExecutor(566): Procedure DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=3 state=RUNNABLE added to the store. 2015-06-17 11:06:48,191 DEBUG [ProcedureExecutorThread-2] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testMasterProcsWithMasterRestarting/write-master:509030000000001 2015-06-17 11:06:48,192 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=3 2015-06-17 11:06:48,195 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=3 2015-06-17 11:06:48,199 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=3 2015-06-17 11:06:48,205 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=3 2015-06-17 11:06:48,214 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=3 2015-06-17 11:06:48,228 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=3 2015-06-17 11:06:48,251 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=3 2015-06-17 11:06:48,295 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=3 2015-06-17 11:06:48,397 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=3 2015-06-17 11:06:48,402 DEBUG [ProcedureExecutorThread-2] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564408402,"tag":[],"qualifier":"state","vlen":2}]},"row":"testMasterProcsWithMasterRestarting"} 2015-06-17 11:06:48,406 INFO [ProcedureExecutorThread-2] hbase.MetaTableAccessor(1692): Updated table testMasterProcsWithMasterRestarting state to DISABLING in META 2015-06-17 11:06:48,501 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=3 2015-06-17 11:06:48,511 INFO [ProcedureExecutorThread-2] procedure.DisableTableProcedure(396): Offlining 4 regions. 2015-06-17 11:06:48,516 DEBUG [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-0] master.AssignmentManager(1300): Starting unassign of testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. (offlining), current state: {c555b4ca922c2de1fdd394fcc2ac20de state=OPEN, ts=1434564407570, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:48,516 DEBUG [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-1] master.AssignmentManager(1300): Starting unassign of testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. (offlining), current state: {3216ee6f55befe8693a09a4e6457d626 state=OPEN, ts=1434564407570, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:48,516 INFO [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-0] master.RegionStates(1072): Transition {c555b4ca922c2de1fdd394fcc2ac20de state=OPEN, ts=1434564407570, server=10.0.0.110,50905,1434564404764} to {c555b4ca922c2de1fdd394fcc2ac20de state=PENDING_CLOSE, ts=1434564408516, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:48,517 INFO [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-1] master.RegionStates(1072): Transition {3216ee6f55befe8693a09a4e6457d626 state=OPEN, ts=1434564407570, server=10.0.0.110,50905,1434564404764} to {3216ee6f55befe8693a09a4e6457d626 state=PENDING_CLOSE, ts=1434564408517, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:48,517 DEBUG [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-2] master.AssignmentManager(1300): Starting unassign of testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. (offlining), current state: {7855514cb57ce135029b4fee0d052405 state=OPEN, ts=1434564407589, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:48,517 INFO [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-1] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. with state=PENDING_CLOSE 2015-06-17 11:06:48,517 INFO [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-0] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. with state=PENDING_CLOSE 2015-06-17 11:06:48,517 DEBUG [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-3] master.AssignmentManager(1300): Starting unassign of testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. (offlining), current state: {3819afeab7512f52d645bda087c8d2eb state=OPEN, ts=1434564407570, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:48,517 INFO [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-2] master.RegionStates(1072): Transition {7855514cb57ce135029b4fee0d052405 state=OPEN, ts=1434564407589, server=10.0.0.110,50905,1434564404764} to {7855514cb57ce135029b4fee0d052405 state=PENDING_CLOSE, ts=1434564408517, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:48,517 INFO [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-3] master.RegionStates(1072): Transition {3819afeab7512f52d645bda087c8d2eb state=OPEN, ts=1434564407570, server=10.0.0.110,50905,1434564404764} to {3819afeab7512f52d645bda087c8d2eb state=PENDING_CLOSE, ts=1434564408517, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:48,518 INFO [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-2] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. with state=PENDING_CLOSE 2015-06-17 11:06:48,519 INFO [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-3] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. with state=PENDING_CLOSE 2015-06-17 11:06:48,524 INFO [PriorityRpcServer.handler=2,queue=0,port=50905] regionserver.RSRpcServices(1082): Close 7855514cb57ce135029b4fee0d052405, moving to null 2015-06-17 11:06:48,524 INFO [PriorityRpcServer.handler=4,queue=0,port=50905] regionserver.RSRpcServices(1082): Close 3819afeab7512f52d645bda087c8d2eb, moving to null 2015-06-17 11:06:48,524 INFO [PriorityRpcServer.handler=6,queue=0,port=50905] regionserver.RSRpcServices(1082): Close 3216ee6f55befe8693a09a4e6457d626, moving to null 2015-06-17 11:06:48,524 INFO [PriorityRpcServer.handler=1,queue=1,port=50905] regionserver.RSRpcServices(1082): Close c555b4ca922c2de1fdd394fcc2ac20de, moving to null 2015-06-17 11:06:48,525 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-1] handler.CloseRegionHandler(90): Processing close of testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. 2015-06-17 11:06:48,525 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-0] handler.CloseRegionHandler(90): Processing close of testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. 2015-06-17 11:06:48,525 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-2] handler.CloseRegionHandler(90): Processing close of testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. 2015-06-17 11:06:48,525 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-1] regionserver.HRegion(1365): Closing testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626.: disabling compactions & flushes 2015-06-17 11:06:48,525 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-1] regionserver.HRegion(1392): Updates disabled for region testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. 2015-06-17 11:06:48,526 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-0] regionserver.HRegion(1365): Closing testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de.: disabling compactions & flushes 2015-06-17 11:06:48,526 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-0] regionserver.HRegion(1392): Updates disabled for region testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. 2015-06-17 11:06:48,526 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-2] regionserver.HRegion(1365): Closing testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405.: disabling compactions & flushes 2015-06-17 11:06:48,526 INFO [StoreCloserThread-testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626.-1] regionserver.HStore(870): Closed f1 2015-06-17 11:06:48,526 INFO [StoreCloserThread-testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de.-1] regionserver.HStore(870): Closed f1 2015-06-17 11:06:48,526 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-2] regionserver.HRegion(1392): Updates disabled for region testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. 2015-06-17 11:06:48,527 INFO [StoreCloserThread-testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405.-1] regionserver.HStore(870): Closed f1 2015-06-17 11:06:48,528 DEBUG [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-2] master.AssignmentManager(884): Sent CLOSE to 10.0.0.110,50905,1434564404764 for region testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. 2015-06-17 11:06:48,528 DEBUG [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-3] master.AssignmentManager(884): Sent CLOSE to 10.0.0.110,50905,1434564404764 for region testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. 2015-06-17 11:06:48,528 DEBUG [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-0] master.AssignmentManager(884): Sent CLOSE to 10.0.0.110,50905,1434564404764 for region testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. 2015-06-17 11:06:48,528 DEBUG [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-1] master.AssignmentManager(884): Sent CLOSE to 10.0.0.110,50905,1434564404764 for region testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. 2015-06-17 11:06:48,535 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-2] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/7855514cb57ce135029b4fee0d052405/recovered.edits/4.seqid to file, newSeqId=4, maxSeqId=2 2015-06-17 11:06:48,536 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/c555b4ca922c2de1fdd394fcc2ac20de/recovered.edits/4.seqid to file, newSeqId=4, maxSeqId=2 2015-06-17 11:06:48,536 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/3216ee6f55befe8693a09a4e6457d626/recovered.edits/4.seqid to file, newSeqId=4, maxSeqId=2 2015-06-17 11:06:48,538 INFO [RS_CLOSE_REGION-10.0.0.110:50905-2] regionserver.HRegion(1486): Closed testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. 2015-06-17 11:06:48,538 INFO [RS_CLOSE_REGION-10.0.0.110:50905-1] regionserver.HRegion(1486): Closed testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. 2015-06-17 11:06:48,538 INFO [RS_CLOSE_REGION-10.0.0.110:50905-0] regionserver.HRegion(1486): Closed testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. 2015-06-17 11:06:48,538 DEBUG [PriorityRpcServer.handler=12,queue=0,port=50903] master.AssignmentManager(2812): Got transition CLOSED for {7855514cb57ce135029b4fee0d052405 state=PENDING_CLOSE, ts=1434564408517, server=10.0.0.110,50905,1434564404764} from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:48,538 DEBUG [PriorityRpcServer.handler=5,queue=1,port=50903] master.AssignmentManager(2812): Got transition CLOSED for {3216ee6f55befe8693a09a4e6457d626 state=PENDING_CLOSE, ts=1434564408517, server=10.0.0.110,50905,1434564404764} from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:48,539 DEBUG [PriorityRpcServer.handler=14,queue=0,port=50903] master.AssignmentManager(2812): Got transition CLOSED for {c555b4ca922c2de1fdd394fcc2ac20de state=PENDING_CLOSE, ts=1434564408516, server=10.0.0.110,50905,1434564404764} from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:48,540 INFO [PriorityRpcServer.handler=12,queue=0,port=50903] master.RegionStates(1072): Transition {7855514cb57ce135029b4fee0d052405 state=PENDING_CLOSE, ts=1434564408517, server=10.0.0.110,50905,1434564404764} to {7855514cb57ce135029b4fee0d052405 state=OFFLINE, ts=1434564408540, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:48,540 INFO [PriorityRpcServer.handler=5,queue=1,port=50903] master.RegionStates(1072): Transition {3216ee6f55befe8693a09a4e6457d626 state=PENDING_CLOSE, ts=1434564408517, server=10.0.0.110,50905,1434564404764} to {3216ee6f55befe8693a09a4e6457d626 state=OFFLINE, ts=1434564408540, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:48,540 INFO [PriorityRpcServer.handler=12,queue=0,port=50903] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. with state=OFFLINE 2015-06-17 11:06:48,540 INFO [PriorityRpcServer.handler=5,queue=1,port=50903] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. with state=OFFLINE 2015-06-17 11:06:48,540 INFO [PriorityRpcServer.handler=14,queue=0,port=50903] master.RegionStates(1072): Transition {c555b4ca922c2de1fdd394fcc2ac20de state=PENDING_CLOSE, ts=1434564408516, server=10.0.0.110,50905,1434564404764} to {c555b4ca922c2de1fdd394fcc2ac20de state=OFFLINE, ts=1434564408540, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:48,541 INFO [PriorityRpcServer.handler=14,queue=0,port=50903] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. with state=OFFLINE 2015-06-17 11:06:48,543 INFO [PriorityRpcServer.handler=12,queue=0,port=50903] master.RegionStates(557): Offlined 7855514cb57ce135029b4fee0d052405 from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:48,543 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-2] handler.CloseRegionHandler(122): Closed testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. 2015-06-17 11:06:48,543 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-2] handler.CloseRegionHandler(90): Processing close of testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. 2015-06-17 11:06:48,544 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-2] regionserver.HRegion(1365): Closing testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb.: disabling compactions & flushes 2015-06-17 11:06:48,544 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-2] regionserver.HRegion(1392): Updates disabled for region testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. 2015-06-17 11:06:48,544 INFO [StoreCloserThread-testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb.-1] regionserver.HStore(870): Closed f1 2015-06-17 11:06:48,544 INFO [PriorityRpcServer.handler=5,queue=1,port=50903] master.RegionStates(557): Offlined 3216ee6f55befe8693a09a4e6457d626 from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:48,545 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-1] handler.CloseRegionHandler(122): Closed testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. 2015-06-17 11:06:48,546 INFO [PriorityRpcServer.handler=14,queue=0,port=50903] master.RegionStates(557): Offlined c555b4ca922c2de1fdd394fcc2ac20de from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:48,547 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-0] handler.CloseRegionHandler(122): Closed testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. 2015-06-17 11:06:48,551 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-2] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/3819afeab7512f52d645bda087c8d2eb/recovered.edits/4.seqid to file, newSeqId=4, maxSeqId=2 2015-06-17 11:06:48,552 INFO [RS_CLOSE_REGION-10.0.0.110:50905-2] regionserver.HRegion(1486): Closed testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. 2015-06-17 11:06:48,553 DEBUG [PriorityRpcServer.handler=18,queue=0,port=50903] master.AssignmentManager(2812): Got transition CLOSED for {3819afeab7512f52d645bda087c8d2eb state=PENDING_CLOSE, ts=1434564408517, server=10.0.0.110,50905,1434564404764} from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:48,554 INFO [PriorityRpcServer.handler=18,queue=0,port=50903] master.RegionStates(1072): Transition {3819afeab7512f52d645bda087c8d2eb state=PENDING_CLOSE, ts=1434564408517, server=10.0.0.110,50905,1434564404764} to {3819afeab7512f52d645bda087c8d2eb state=OFFLINE, ts=1434564408554, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:48,554 INFO [PriorityRpcServer.handler=18,queue=0,port=50903] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. with state=OFFLINE 2015-06-17 11:06:48,556 INFO [PriorityRpcServer.handler=18,queue=0,port=50903] master.RegionStates(557): Offlined 3819afeab7512f52d645bda087c8d2eb from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:48,557 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-2] handler.CloseRegionHandler(122): Closed testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. 2015-06-17 11:06:48,605 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=3 2015-06-17 11:06:48,672 DEBUG [ProcedureExecutorThread-2] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564408672,"tag":[],"qualifier":"state","vlen":2}]},"row":"testMasterProcsWithMasterRestarting"} 2015-06-17 11:06:48,675 INFO [ProcedureExecutorThread-2] hbase.MetaTableAccessor(1692): Updated table testMasterProcsWithMasterRestarting state to DISABLED in META 2015-06-17 11:06:48,676 INFO [ProcedureExecutorThread-2] procedure.DisableTableProcedure(425): Disabled table, testMasterProcsWithMasterRestarting, is completed. 2015-06-17 11:06:48,708 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=3 2015-06-17 11:06:48,886 DEBUG [ProcedureExecutorThread-2] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testMasterProcsWithMasterRestarting/write-master:509030000000001 2015-06-17 11:06:48,886 DEBUG [ProcedureExecutorThread-2] procedure2.ProcedureExecutor(729): Procedure completed in 703msec: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=3 state=FINISHED 2015-06-17 11:06:48,912 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=3 2015-06-17 11:06:48,913 INFO [Thread-199] client.HBaseAdmin$TableFuture(4327): Operation: DISABLE, Table Name: default:testMasterProcsWithMasterRestarting completed 2015-06-17 11:06:48,914 INFO [Thread-199] procedure.TestMasterRestartWithProcedures$3(129): executing enable table step=0 2015-06-17 11:06:48,916 INFO [Thread-199] client.HBaseAdmin$8(994): Started enable of testMasterProcsWithMasterRestarting 2015-06-17 11:06:48,921 INFO [B.defaultRpcServer.handler=3,queue=0,port=50903] master.HMaster(1717): Client=ndimiduk/null enable testMasterProcsWithMasterRestarting 2015-06-17 11:06:49,031 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50903] procedure2.ProcedureExecutor(566): Procedure EnableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=4 state=RUNNABLE added to the store. 2015-06-17 11:06:49,034 DEBUG [ProcedureExecutorThread-3] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testMasterProcsWithMasterRestarting/write-master:509030000000002 2015-06-17 11:06:49,035 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=4 2015-06-17 11:06:49,038 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=4 2015-06-17 11:06:49,042 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=4 2015-06-17 11:06:49,047 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=4 2015-06-17 11:06:49,054 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=4 2015-06-17 11:06:49,067 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=4 2015-06-17 11:06:49,091 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=4 2015-06-17 11:06:49,135 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=4 2015-06-17 11:06:49,238 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=4 2015-06-17 11:06:49,246 INFO [ProcedureExecutorThread-3] procedure.EnableTableProcedure(352): Attempting to enable the table testMasterProcsWithMasterRestarting 2015-06-17 11:06:49,247 DEBUG [ProcedureExecutorThread-3] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564409247,"tag":[],"qualifier":"state","vlen":2}]},"row":"testMasterProcsWithMasterRestarting"} 2015-06-17 11:06:49,252 INFO [ProcedureExecutorThread-3] hbase.MetaTableAccessor(1692): Updated table testMasterProcsWithMasterRestarting state to ENABLING in META 2015-06-17 11:06:49,340 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=4 2015-06-17 11:06:49,364 INFO [ProcedureExecutorThread-3] procedure.EnableTableProcedure(444): Table 'testMasterProcsWithMasterRestarting' has 4 regions, of which 4 are offline. 2015-06-17 11:06:49,364 INFO [ProcedureExecutorThread-3] procedure.EnableTableProcedure(455): Bulk assigning 4 region(s) across 2 server(s), retainAssignment=true 2015-06-17 11:06:49,366 INFO [10.0.0.110,50903,1434564404489-GeneralBulkAssigner-1] master.AssignmentManager(680): Assigning 4 region(s) to 10.0.0.110,50905,1434564404764 2015-06-17 11:06:49,366 DEBUG [ProcedureExecutorThread-3] master.GeneralBulkAssigner(152): Timeout-on-RIT=124000 2015-06-17 11:06:49,381 INFO [10.0.0.110,50903,1434564404489-GeneralBulkAssigner-1] master.RegionStates(1072): Transition {c555b4ca922c2de1fdd394fcc2ac20de state=OFFLINE, ts=1434564408540, server=10.0.0.110,50905,1434564404764} to {c555b4ca922c2de1fdd394fcc2ac20de state=PENDING_OPEN, ts=1434564409381, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:49,381 INFO [10.0.0.110,50903,1434564404489-GeneralBulkAssigner-1] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. with state=PENDING_OPEN 2015-06-17 11:06:49,382 INFO [10.0.0.110,50903,1434564404489-GeneralBulkAssigner-1] master.RegionStates(1072): Transition {3216ee6f55befe8693a09a4e6457d626 state=OFFLINE, ts=1434564408540, server=10.0.0.110,50905,1434564404764} to {3216ee6f55befe8693a09a4e6457d626 state=PENDING_OPEN, ts=1434564409382, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:49,382 INFO [10.0.0.110,50903,1434564404489-GeneralBulkAssigner-1] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. with state=PENDING_OPEN 2015-06-17 11:06:49,383 INFO [10.0.0.110,50903,1434564404489-GeneralBulkAssigner-1] master.RegionStates(1072): Transition {7855514cb57ce135029b4fee0d052405 state=OFFLINE, ts=1434564408540, server=10.0.0.110,50905,1434564404764} to {7855514cb57ce135029b4fee0d052405 state=PENDING_OPEN, ts=1434564409383, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:49,383 INFO [10.0.0.110,50903,1434564404489-GeneralBulkAssigner-1] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. with state=PENDING_OPEN 2015-06-17 11:06:49,384 INFO [10.0.0.110,50903,1434564404489-GeneralBulkAssigner-1] master.RegionStates(1072): Transition {3819afeab7512f52d645bda087c8d2eb state=OFFLINE, ts=1434564408554, server=10.0.0.110,50905,1434564404764} to {3819afeab7512f52d645bda087c8d2eb state=PENDING_OPEN, ts=1434564409384, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:49,384 INFO [10.0.0.110,50903,1434564404489-GeneralBulkAssigner-1] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. with state=PENDING_OPEN 2015-06-17 11:06:49,386 INFO [PriorityRpcServer.handler=12,queue=0,port=50905] regionserver.RSRpcServices(1432): Open testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. 2015-06-17 11:06:49,391 INFO [PriorityRpcServer.handler=12,queue=0,port=50905] regionserver.RSRpcServices(1432): Open testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. 2015-06-17 11:06:49,391 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-1] regionserver.HRegion(6070): Opening region: {ENCODED => c555b4ca922c2de1fdd394fcc2ac20de, NAME => 'testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de.', STARTKEY => '', ENDKEY => 'a'} 2015-06-17 11:06:49,391 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testMasterProcsWithMasterRestarting c555b4ca922c2de1fdd394fcc2ac20de 2015-06-17 11:06:49,391 INFO [PriorityRpcServer.handler=12,queue=0,port=50905] regionserver.RSRpcServices(1432): Open testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. 2015-06-17 11:06:49,391 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-1] regionserver.HRegion(722): Instantiated testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. 2015-06-17 11:06:49,392 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-2] regionserver.HRegion(6070): Opening region: {ENCODED => 3216ee6f55befe8693a09a4e6457d626, NAME => 'testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626.', STARTKEY => 'a', ENDKEY => 'b'} 2015-06-17 11:06:49,392 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-2] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testMasterProcsWithMasterRestarting 3216ee6f55befe8693a09a4e6457d626 2015-06-17 11:06:49,392 INFO [PriorityRpcServer.handler=12,queue=0,port=50905] regionserver.RSRpcServices(1432): Open testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. 2015-06-17 11:06:49,392 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-2] regionserver.HRegion(722): Instantiated testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. 2015-06-17 11:06:49,392 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.HRegion(6070): Opening region: {ENCODED => 7855514cb57ce135029b4fee0d052405, NAME => 'testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405.', STARTKEY => 'b', ENDKEY => 'c'} 2015-06-17 11:06:49,393 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testMasterProcsWithMasterRestarting 7855514cb57ce135029b4fee0d052405 2015-06-17 11:06:49,393 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.HRegion(722): Instantiated testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. 2015-06-17 11:06:49,395 INFO [StoreOpener-c555b4ca922c2de1fdd394fcc2ac20de-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:06:49,395 INFO [StoreOpener-3216ee6f55befe8693a09a4e6457d626-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:06:49,395 INFO [StoreOpener-c555b4ca922c2de1fdd394fcc2ac20de-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:06:49,396 INFO [StoreOpener-3216ee6f55befe8693a09a4e6457d626-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:06:49,396 INFO [StoreOpener-7855514cb57ce135029b4fee0d052405-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:06:49,397 INFO [StoreOpener-7855514cb57ce135029b4fee0d052405-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:06:49,397 DEBUG [StoreOpener-c555b4ca922c2de1fdd394fcc2ac20de-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/c555b4ca922c2de1fdd394fcc2ac20de/f1 2015-06-17 11:06:49,397 DEBUG [StoreOpener-3216ee6f55befe8693a09a4e6457d626-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/3216ee6f55befe8693a09a4e6457d626/f1 2015-06-17 11:06:49,398 DEBUG [StoreOpener-7855514cb57ce135029b4fee0d052405-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/7855514cb57ce135029b4fee0d052405/f1 2015-06-17 11:06:49,400 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-1] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/c555b4ca922c2de1fdd394fcc2ac20de 2015-06-17 11:06:49,400 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-2] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/3216ee6f55befe8693a09a4e6457d626 2015-06-17 11:06:49,401 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/7855514cb57ce135029b4fee0d052405 2015-06-17 11:06:49,403 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-1] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:06:49,403 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-2] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:06:49,404 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:06:49,408 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/c555b4ca922c2de1fdd394fcc2ac20de/recovered.edits/5.seqid to file, newSeqId=5, maxSeqId=4 2015-06-17 11:06:49,408 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-2] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/3216ee6f55befe8693a09a4e6457d626/recovered.edits/5.seqid to file, newSeqId=5, maxSeqId=4 2015-06-17 11:06:49,409 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/7855514cb57ce135029b4fee0d052405/recovered.edits/5.seqid to file, newSeqId=5, maxSeqId=4 2015-06-17 11:06:49,410 INFO [RS_OPEN_REGION-10.0.0.110:50905-1] regionserver.HRegion(848): Onlined c555b4ca922c2de1fdd394fcc2ac20de; next sequenceid=5 2015-06-17 11:06:49,410 INFO [RS_OPEN_REGION-10.0.0.110:50905-2] regionserver.HRegion(848): Onlined 3216ee6f55befe8693a09a4e6457d626; next sequenceid=5 2015-06-17 11:06:49,411 INFO [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.HRegion(848): Onlined 7855514cb57ce135029b4fee0d052405; next sequenceid=5 2015-06-17 11:06:49,412 INFO [PostOpenDeployTasks:3216ee6f55befe8693a09a4e6457d626] regionserver.HRegionServer(1893): Post open deploy tasks for testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. 2015-06-17 11:06:49,412 INFO [PostOpenDeployTasks:7855514cb57ce135029b4fee0d052405] regionserver.HRegionServer(1893): Post open deploy tasks for testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. 2015-06-17 11:06:49,412 INFO [PostOpenDeployTasks:c555b4ca922c2de1fdd394fcc2ac20de] regionserver.HRegionServer(1893): Post open deploy tasks for testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. 2015-06-17 11:06:49,413 DEBUG [PriorityRpcServer.handler=5,queue=1,port=50903] master.AssignmentManager(2812): Got transition OPENED for {3216ee6f55befe8693a09a4e6457d626 state=PENDING_OPEN, ts=1434564409382, server=10.0.0.110,50905,1434564404764} from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:49,413 DEBUG [PriorityRpcServer.handler=0,queue=0,port=50903] master.AssignmentManager(2812): Got transition OPENED for {7855514cb57ce135029b4fee0d052405 state=PENDING_OPEN, ts=1434564409383, server=10.0.0.110,50905,1434564404764} from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:49,413 INFO [PriorityRpcServer.handler=5,queue=1,port=50903] master.RegionStates(1072): Transition {3216ee6f55befe8693a09a4e6457d626 state=PENDING_OPEN, ts=1434564409382, server=10.0.0.110,50905,1434564404764} to {3216ee6f55befe8693a09a4e6457d626 state=OPEN, ts=1434564409413, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:49,413 INFO [PriorityRpcServer.handler=0,queue=0,port=50903] master.RegionStates(1072): Transition {7855514cb57ce135029b4fee0d052405 state=PENDING_OPEN, ts=1434564409383, server=10.0.0.110,50905,1434564404764} to {7855514cb57ce135029b4fee0d052405 state=OPEN, ts=1434564409413, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:49,413 DEBUG [PriorityRpcServer.handler=12,queue=0,port=50903] master.AssignmentManager(2812): Got transition OPENED for {c555b4ca922c2de1fdd394fcc2ac20de state=PENDING_OPEN, ts=1434564409381, server=10.0.0.110,50905,1434564404764} from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:49,414 INFO [PriorityRpcServer.handler=0,queue=0,port=50903] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. with state=OPEN&openSeqNum=5&server=10.0.0.110,50905,1434564404764 2015-06-17 11:06:49,414 INFO [PriorityRpcServer.handler=5,queue=1,port=50903] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. with state=OPEN&openSeqNum=5&server=10.0.0.110,50905,1434564404764 2015-06-17 11:06:49,414 INFO [PriorityRpcServer.handler=12,queue=0,port=50903] master.RegionStates(1072): Transition {c555b4ca922c2de1fdd394fcc2ac20de state=PENDING_OPEN, ts=1434564409381, server=10.0.0.110,50905,1434564404764} to {c555b4ca922c2de1fdd394fcc2ac20de state=OPEN, ts=1434564409414, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:49,414 INFO [PriorityRpcServer.handler=12,queue=0,port=50903] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. with state=OPEN&openSeqNum=5&server=10.0.0.110,50905,1434564404764 2015-06-17 11:06:49,416 DEBUG [PriorityRpcServer.handler=0,queue=0,port=50903] master.RegionStates(417): Onlined 7855514cb57ce135029b4fee0d052405 on 10.0.0.110,50905,1434564404764 2015-06-17 11:06:49,416 DEBUG [PriorityRpcServer.handler=5,queue=1,port=50903] master.RegionStates(417): Onlined 3216ee6f55befe8693a09a4e6457d626 on 10.0.0.110,50905,1434564404764 2015-06-17 11:06:49,416 DEBUG [PriorityRpcServer.handler=12,queue=0,port=50903] master.RegionStates(417): Onlined c555b4ca922c2de1fdd394fcc2ac20de on 10.0.0.110,50905,1434564404764 2015-06-17 11:06:49,418 DEBUG [PostOpenDeployTasks:7855514cb57ce135029b4fee0d052405] regionserver.HRegionServer(1920): Finished post open deploy task for testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. 2015-06-17 11:06:49,419 DEBUG [PostOpenDeployTasks:3216ee6f55befe8693a09a4e6457d626] regionserver.HRegionServer(1920): Finished post open deploy task for testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. 2015-06-17 11:06:49,419 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] handler.OpenRegionHandler(125): Opened testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. on 10.0.0.110,50905,1434564404764 2015-06-17 11:06:49,419 DEBUG [PostOpenDeployTasks:c555b4ca922c2de1fdd394fcc2ac20de] regionserver.HRegionServer(1920): Finished post open deploy task for testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. 2015-06-17 11:06:49,419 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-2] handler.OpenRegionHandler(125): Opened testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. on 10.0.0.110,50905,1434564404764 2015-06-17 11:06:49,419 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-1] handler.OpenRegionHandler(125): Opened testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. on 10.0.0.110,50905,1434564404764 2015-06-17 11:06:49,419 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.HRegion(6070): Opening region: {ENCODED => 3819afeab7512f52d645bda087c8d2eb, NAME => 'testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb.', STARTKEY => 'c', ENDKEY => ''} 2015-06-17 11:06:49,420 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testMasterProcsWithMasterRestarting 3819afeab7512f52d645bda087c8d2eb 2015-06-17 11:06:49,420 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.HRegion(722): Instantiated testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. 2015-06-17 11:06:49,423 INFO [StoreOpener-3819afeab7512f52d645bda087c8d2eb-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:06:49,424 INFO [StoreOpener-3819afeab7512f52d645bda087c8d2eb-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:06:49,425 DEBUG [StoreOpener-3819afeab7512f52d645bda087c8d2eb-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/3819afeab7512f52d645bda087c8d2eb/f1 2015-06-17 11:06:49,427 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/3819afeab7512f52d645bda087c8d2eb 2015-06-17 11:06:49,430 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:06:49,435 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/3819afeab7512f52d645bda087c8d2eb/recovered.edits/5.seqid to file, newSeqId=5, maxSeqId=4 2015-06-17 11:06:49,436 INFO [RS_OPEN_REGION-10.0.0.110:50905-0] regionserver.HRegion(848): Onlined 3819afeab7512f52d645bda087c8d2eb; next sequenceid=5 2015-06-17 11:06:49,437 INFO [PostOpenDeployTasks:3819afeab7512f52d645bda087c8d2eb] regionserver.HRegionServer(1893): Post open deploy tasks for testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. 2015-06-17 11:06:49,438 DEBUG [PriorityRpcServer.handler=19,queue=1,port=50903] master.AssignmentManager(2812): Got transition OPENED for {3819afeab7512f52d645bda087c8d2eb state=PENDING_OPEN, ts=1434564409384, server=10.0.0.110,50905,1434564404764} from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:49,438 INFO [PriorityRpcServer.handler=19,queue=1,port=50903] master.RegionStates(1072): Transition {3819afeab7512f52d645bda087c8d2eb state=PENDING_OPEN, ts=1434564409384, server=10.0.0.110,50905,1434564404764} to {3819afeab7512f52d645bda087c8d2eb state=OPEN, ts=1434564409438, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:49,438 INFO [PriorityRpcServer.handler=19,queue=1,port=50903] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. with state=OPEN&openSeqNum=5&server=10.0.0.110,50905,1434564404764 2015-06-17 11:06:49,439 DEBUG [PriorityRpcServer.handler=19,queue=1,port=50903] master.RegionStates(417): Onlined 3819afeab7512f52d645bda087c8d2eb on 10.0.0.110,50905,1434564404764 2015-06-17 11:06:49,440 DEBUG [10.0.0.110,50903,1434564404489-GeneralBulkAssigner-1] master.AssignmentManager(851): Bulk assigning done for 10.0.0.110,50905,1434564404764 2015-06-17 11:06:49,440 DEBUG [ProcedureExecutorThread-3] master.GeneralBulkAssigner(128): bulk assigning total 4 regions to 2 servers, took 73ms, successfully 2015-06-17 11:06:49,441 DEBUG [PostOpenDeployTasks:3819afeab7512f52d645bda087c8d2eb] regionserver.HRegionServer(1920): Finished post open deploy task for testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. 2015-06-17 11:06:49,441 DEBUG [RS_OPEN_REGION-10.0.0.110:50905-0] handler.OpenRegionHandler(125): Opened testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. on 10.0.0.110,50905,1434564404764 2015-06-17 11:06:49,442 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=4 2015-06-17 11:06:49,544 DEBUG [ProcedureExecutorThread-3] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564409544,"tag":[],"qualifier":"state","vlen":2}]},"row":"testMasterProcsWithMasterRestarting"} 2015-06-17 11:06:49,545 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=4 2015-06-17 11:06:49,548 INFO [ProcedureExecutorThread-3] hbase.MetaTableAccessor(1692): Updated table testMasterProcsWithMasterRestarting state to ENABLED in META 2015-06-17 11:06:49,548 INFO [ProcedureExecutorThread-3] procedure.EnableTableProcedure(501): Table 'testMasterProcsWithMasterRestarting' was successfully enabled. 2015-06-17 11:06:49,749 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=4 2015-06-17 11:06:49,757 DEBUG [ProcedureExecutorThread-3] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testMasterProcsWithMasterRestarting/write-master:509030000000002 2015-06-17 11:06:49,757 DEBUG [ProcedureExecutorThread-3] procedure2.ProcedureExecutor(729): Procedure completed in 730msec: EnableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=4 state=FINISHED 2015-06-17 11:06:49,953 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=4 2015-06-17 11:06:49,954 INFO [Thread-199] client.HBaseAdmin$TableFuture(4327): Operation: ENABLE, Table Name: default:testMasterProcsWithMasterRestarting completed 2015-06-17 11:06:49,955 INFO [Thread-199] procedure.TestMasterRestartWithProcedures$4(139): executing disable table step=0 2015-06-17 11:06:49,956 INFO [Thread-199] client.HBaseAdmin$9(1131): Started disable of testMasterProcsWithMasterRestarting 2015-06-17 11:06:49,957 INFO [B.defaultRpcServer.handler=3,queue=0,port=50903] master.HMaster(1743): Client=ndimiduk/null disable testMasterProcsWithMasterRestarting 2015-06-17 11:06:50,060 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50903] procedure2.ProcedureExecutor(566): Procedure DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=5 state=RUNNABLE added to the store. 2015-06-17 11:06:50,062 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=5 2015-06-17 11:06:50,063 DEBUG [ProcedureExecutorThread-4] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testMasterProcsWithMasterRestarting/write-master:509030000000003 2015-06-17 11:06:50,065 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=5 2015-06-17 11:06:50,069 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=5 2015-06-17 11:06:50,073 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=5 2015-06-17 11:06:50,082 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=5 2015-06-17 11:06:50,095 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=5 2015-06-17 11:06:50,118 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=5 2015-06-17 11:06:50,161 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=5 2015-06-17 11:06:50,264 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=5 2015-06-17 11:06:50,273 DEBUG [ProcedureExecutorThread-4] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564410272,"tag":[],"qualifier":"state","vlen":2}]},"row":"testMasterProcsWithMasterRestarting"} 2015-06-17 11:06:50,328 INFO [ProcedureExecutorThread-4] hbase.MetaTableAccessor(1692): Updated table testMasterProcsWithMasterRestarting state to DISABLING in META 2015-06-17 11:06:50,367 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=5 2015-06-17 11:06:50,431 INFO [ProcedureExecutorThread-4] procedure.DisableTableProcedure(396): Offlining 4 regions. 2015-06-17 11:06:50,433 DEBUG [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-0] master.AssignmentManager(1300): Starting unassign of testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. (offlining), current state: {c555b4ca922c2de1fdd394fcc2ac20de state=OPEN, ts=1434564409414, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:50,433 INFO [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-0] master.RegionStates(1072): Transition {c555b4ca922c2de1fdd394fcc2ac20de state=OPEN, ts=1434564409414, server=10.0.0.110,50905,1434564404764} to {c555b4ca922c2de1fdd394fcc2ac20de state=PENDING_CLOSE, ts=1434564410433, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:50,433 DEBUG [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-1] master.AssignmentManager(1300): Starting unassign of testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. (offlining), current state: {3216ee6f55befe8693a09a4e6457d626 state=OPEN, ts=1434564409413, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:50,433 INFO [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-0] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. with state=PENDING_CLOSE 2015-06-17 11:06:50,433 INFO [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-1] master.RegionStates(1072): Transition {3216ee6f55befe8693a09a4e6457d626 state=OPEN, ts=1434564409413, server=10.0.0.110,50905,1434564404764} to {3216ee6f55befe8693a09a4e6457d626 state=PENDING_CLOSE, ts=1434564410433, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:50,434 DEBUG [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-2] master.AssignmentManager(1300): Starting unassign of testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. (offlining), current state: {7855514cb57ce135029b4fee0d052405 state=OPEN, ts=1434564409413, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:50,434 INFO [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-1] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. with state=PENDING_CLOSE 2015-06-17 11:06:50,434 DEBUG [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-3] master.AssignmentManager(1300): Starting unassign of testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. (offlining), current state: {3819afeab7512f52d645bda087c8d2eb state=OPEN, ts=1434564409438, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:50,434 INFO [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-2] master.RegionStates(1072): Transition {7855514cb57ce135029b4fee0d052405 state=OPEN, ts=1434564409413, server=10.0.0.110,50905,1434564404764} to {7855514cb57ce135029b4fee0d052405 state=PENDING_CLOSE, ts=1434564410434, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:50,434 INFO [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-3] master.RegionStates(1072): Transition {3819afeab7512f52d645bda087c8d2eb state=OPEN, ts=1434564409438, server=10.0.0.110,50905,1434564404764} to {3819afeab7512f52d645bda087c8d2eb state=PENDING_CLOSE, ts=1434564410434, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:50,434 INFO [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-2] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. with state=PENDING_CLOSE 2015-06-17 11:06:50,434 INFO [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-3] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. with state=PENDING_CLOSE 2015-06-17 11:06:50,436 INFO [PriorityRpcServer.handler=7,queue=1,port=50905] regionserver.RSRpcServices(1082): Close c555b4ca922c2de1fdd394fcc2ac20de, moving to null 2015-06-17 11:06:50,436 INFO [PriorityRpcServer.handler=9,queue=1,port=50905] regionserver.RSRpcServices(1082): Close 3216ee6f55befe8693a09a4e6457d626, moving to null 2015-06-17 11:06:50,436 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-1] handler.CloseRegionHandler(90): Processing close of testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. 2015-06-17 11:06:50,436 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-0] handler.CloseRegionHandler(90): Processing close of testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. 2015-06-17 11:06:50,437 INFO [PriorityRpcServer.handler=13,queue=1,port=50905] regionserver.RSRpcServices(1082): Close 3819afeab7512f52d645bda087c8d2eb, moving to null 2015-06-17 11:06:50,436 INFO [PriorityRpcServer.handler=11,queue=1,port=50905] regionserver.RSRpcServices(1082): Close 7855514cb57ce135029b4fee0d052405, moving to null 2015-06-17 11:06:50,437 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-2] handler.CloseRegionHandler(90): Processing close of testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. 2015-06-17 11:06:50,437 DEBUG [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-0] master.AssignmentManager(884): Sent CLOSE to 10.0.0.110,50905,1434564404764 for region testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. 2015-06-17 11:06:50,437 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-1] regionserver.HRegion(1365): Closing testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de.: disabling compactions & flushes 2015-06-17 11:06:50,438 DEBUG [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-3] master.AssignmentManager(884): Sent CLOSE to 10.0.0.110,50905,1434564404764 for region testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. 2015-06-17 11:06:50,437 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-0] regionserver.HRegion(1365): Closing testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626.: disabling compactions & flushes 2015-06-17 11:06:50,438 DEBUG [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-2] master.AssignmentManager(884): Sent CLOSE to 10.0.0.110,50905,1434564404764 for region testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. 2015-06-17 11:06:50,438 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-1] regionserver.HRegion(1392): Updates disabled for region testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. 2015-06-17 11:06:50,437 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-2] regionserver.HRegion(1365): Closing testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb.: disabling compactions & flushes 2015-06-17 11:06:50,437 DEBUG [10.0.0.110,50903,1434564404489-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-1] master.AssignmentManager(884): Sent CLOSE to 10.0.0.110,50905,1434564404764 for region testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. 2015-06-17 11:06:50,438 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-2] regionserver.HRegion(1392): Updates disabled for region testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. 2015-06-17 11:06:50,438 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-0] regionserver.HRegion(1392): Updates disabled for region testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. 2015-06-17 11:06:50,438 INFO [StoreCloserThread-testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de.-1] regionserver.HStore(870): Closed f1 2015-06-17 11:06:50,439 INFO [StoreCloserThread-testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb.-1] regionserver.HStore(870): Closed f1 2015-06-17 11:06:50,439 INFO [StoreCloserThread-testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626.-1] regionserver.HStore(870): Closed f1 2015-06-17 11:06:50,446 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/3216ee6f55befe8693a09a4e6457d626/recovered.edits/7.seqid to file, newSeqId=7, maxSeqId=5 2015-06-17 11:06:50,446 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-2] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/3819afeab7512f52d645bda087c8d2eb/recovered.edits/7.seqid to file, newSeqId=7, maxSeqId=5 2015-06-17 11:06:50,446 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/c555b4ca922c2de1fdd394fcc2ac20de/recovered.edits/7.seqid to file, newSeqId=7, maxSeqId=5 2015-06-17 11:06:50,447 INFO [RS_CLOSE_REGION-10.0.0.110:50905-0] regionserver.HRegion(1486): Closed testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. 2015-06-17 11:06:50,447 INFO [RS_CLOSE_REGION-10.0.0.110:50905-2] regionserver.HRegion(1486): Closed testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. 2015-06-17 11:06:50,447 INFO [RS_CLOSE_REGION-10.0.0.110:50905-1] regionserver.HRegion(1486): Closed testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. 2015-06-17 11:06:50,447 DEBUG [PriorityRpcServer.handler=7,queue=1,port=50903] master.AssignmentManager(2812): Got transition CLOSED for {3216ee6f55befe8693a09a4e6457d626 state=PENDING_CLOSE, ts=1434564410433, server=10.0.0.110,50905,1434564404764} from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:50,448 DEBUG [PriorityRpcServer.handler=6,queue=0,port=50903] master.AssignmentManager(2812): Got transition CLOSED for {3819afeab7512f52d645bda087c8d2eb state=PENDING_CLOSE, ts=1434564410434, server=10.0.0.110,50905,1434564404764} from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:50,448 DEBUG [PriorityRpcServer.handler=9,queue=1,port=50903] master.AssignmentManager(2812): Got transition CLOSED for {c555b4ca922c2de1fdd394fcc2ac20de state=PENDING_CLOSE, ts=1434564410433, server=10.0.0.110,50905,1434564404764} from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:50,449 INFO [PriorityRpcServer.handler=7,queue=1,port=50903] master.RegionStates(1072): Transition {3216ee6f55befe8693a09a4e6457d626 state=PENDING_CLOSE, ts=1434564410433, server=10.0.0.110,50905,1434564404764} to {3216ee6f55befe8693a09a4e6457d626 state=OFFLINE, ts=1434564410449, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:50,449 INFO [PriorityRpcServer.handler=6,queue=0,port=50903] master.RegionStates(1072): Transition {3819afeab7512f52d645bda087c8d2eb state=PENDING_CLOSE, ts=1434564410434, server=10.0.0.110,50905,1434564404764} to {3819afeab7512f52d645bda087c8d2eb state=OFFLINE, ts=1434564410449, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:50,449 INFO [PriorityRpcServer.handler=7,queue=1,port=50903] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. with state=OFFLINE 2015-06-17 11:06:50,449 INFO [PriorityRpcServer.handler=9,queue=1,port=50903] master.RegionStates(1072): Transition {c555b4ca922c2de1fdd394fcc2ac20de state=PENDING_CLOSE, ts=1434564410433, server=10.0.0.110,50905,1434564404764} to {c555b4ca922c2de1fdd394fcc2ac20de state=OFFLINE, ts=1434564410449, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:50,449 INFO [PriorityRpcServer.handler=6,queue=0,port=50903] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. with state=OFFLINE 2015-06-17 11:06:50,449 INFO [PriorityRpcServer.handler=9,queue=1,port=50903] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. with state=OFFLINE 2015-06-17 11:06:50,452 INFO [PriorityRpcServer.handler=7,queue=1,port=50903] master.RegionStates(557): Offlined 3216ee6f55befe8693a09a4e6457d626 from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:50,452 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-0] handler.CloseRegionHandler(122): Closed testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. 2015-06-17 11:06:50,452 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-0] handler.CloseRegionHandler(90): Processing close of testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. 2015-06-17 11:06:50,452 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-0] regionserver.HRegion(1365): Closing testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405.: disabling compactions & flushes 2015-06-17 11:06:50,453 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-0] regionserver.HRegion(1392): Updates disabled for region testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. 2015-06-17 11:06:50,453 INFO [StoreCloserThread-testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405.-1] regionserver.HStore(870): Closed f1 2015-06-17 11:06:50,453 INFO [PriorityRpcServer.handler=9,queue=1,port=50903] master.RegionStates(557): Offlined c555b4ca922c2de1fdd394fcc2ac20de from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:50,453 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-1] handler.CloseRegionHandler(122): Closed testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. 2015-06-17 11:06:50,454 INFO [PriorityRpcServer.handler=6,queue=0,port=50903] master.RegionStates(557): Offlined 3819afeab7512f52d645bda087c8d2eb from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:50,455 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-2] handler.CloseRegionHandler(122): Closed testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. 2015-06-17 11:06:50,460 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/default/testMasterProcsWithMasterRestarting/7855514cb57ce135029b4fee0d052405/recovered.edits/7.seqid to file, newSeqId=7, maxSeqId=5 2015-06-17 11:06:50,461 INFO [RS_CLOSE_REGION-10.0.0.110:50905-0] regionserver.HRegion(1486): Closed testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. 2015-06-17 11:06:50,461 DEBUG [PriorityRpcServer.handler=17,queue=1,port=50903] master.AssignmentManager(2812): Got transition CLOSED for {7855514cb57ce135029b4fee0d052405 state=PENDING_CLOSE, ts=1434564410434, server=10.0.0.110,50905,1434564404764} from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:50,463 INFO [PriorityRpcServer.handler=17,queue=1,port=50903] master.RegionStates(1072): Transition {7855514cb57ce135029b4fee0d052405 state=PENDING_CLOSE, ts=1434564410434, server=10.0.0.110,50905,1434564404764} to {7855514cb57ce135029b4fee0d052405 state=OFFLINE, ts=1434564410463, server=10.0.0.110,50905,1434564404764} 2015-06-17 11:06:50,463 INFO [PriorityRpcServer.handler=17,queue=1,port=50903] master.RegionStateStore(207): Updating row testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. with state=OFFLINE 2015-06-17 11:06:50,465 INFO [PriorityRpcServer.handler=17,queue=1,port=50903] master.RegionStates(557): Offlined 7855514cb57ce135029b4fee0d052405 from 10.0.0.110,50905,1434564404764 2015-06-17 11:06:50,465 DEBUG [RS_CLOSE_REGION-10.0.0.110:50905-0] handler.CloseRegionHandler(122): Closed testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. 2015-06-17 11:06:50,469 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=5 2015-06-17 11:06:50,573 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=5 2015-06-17 11:06:50,588 DEBUG [ProcedureExecutorThread-4] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564410588,"tag":[],"qualifier":"state","vlen":2}]},"row":"testMasterProcsWithMasterRestarting"} 2015-06-17 11:06:50,592 INFO [ProcedureExecutorThread-4] hbase.MetaTableAccessor(1692): Updated table testMasterProcsWithMasterRestarting state to DISABLED in META 2015-06-17 11:06:50,593 INFO [ProcedureExecutorThread-4] procedure.DisableTableProcedure(425): Disabled table, testMasterProcsWithMasterRestarting, is completed. 2015-06-17 11:06:50,777 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=5 2015-06-17 11:06:50,802 DEBUG [ProcedureExecutorThread-4] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testMasterProcsWithMasterRestarting/write-master:509030000000003 2015-06-17 11:06:50,802 DEBUG [ProcedureExecutorThread-4] procedure2.ProcedureExecutor(729): Procedure completed in 738msec: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=5 state=FINISHED 2015-06-17 11:06:50,980 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=5 2015-06-17 11:06:50,981 INFO [Thread-199] client.HBaseAdmin$TableFuture(4327): Operation: DISABLE, Table Name: default:testMasterProcsWithMasterRestarting completed 2015-06-17 11:06:50,981 INFO [Thread-199] procedure.TestMasterRestartWithProcedures(146): executing delete table step=0 2015-06-17 11:06:50,989 INFO [B.defaultRpcServer.handler=3,queue=0,port=50903] master.HMaster(1611): Client=ndimiduk/null delete testMasterProcsWithMasterRestarting 2015-06-17 11:06:51,100 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50903] procedure2.ProcedureExecutor(566): Procedure DeleteTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=6 state=RUNNABLE added to the store. 2015-06-17 11:06:51,103 DEBUG [ProcedureExecutorThread-5] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testMasterProcsWithMasterRestarting/write-master:509030000000004 2015-06-17 11:06:51,104 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=6 2015-06-17 11:06:51,107 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=6 2015-06-17 11:06:51,107 DEBUG [ProcedureExecutorThread-5] procedure.DeleteTableProcedure(110): waiting for 'testMasterProcsWithMasterRestarting' regions in transition 2015-06-17 11:06:51,111 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=6 2015-06-17 11:06:51,116 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=6 2015-06-17 11:06:51,123 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=6 2015-06-17 11:06:51,136 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=6 2015-06-17 11:06:51,160 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=6 2015-06-17 11:06:51,204 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=6 2015-06-17 11:06:51,221 DEBUG [ProcedureExecutorThread-5] procedure.DeleteTableProcedure(121): delete 'testMasterProcsWithMasterRestarting' regions from META 2015-06-17 11:06:51,223 DEBUG [ProcedureExecutorThread-5] hbase.MetaTableAccessor(1403): Delete{"ts":9223372036854775807,"totalColumns":1,"families":{"info":[{"timestamp":1434564411222,"tag":[],"qualifier":"","vlen":0}]},"row":"testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de."}, Delete{"ts":9223372036854775807,"totalColumns":1,"families":{"info":[{"timestamp":1434564411222,"tag":[],"qualifier":"","vlen":0}]},"row":"testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626."}, Delete{"ts":9223372036854775807,"totalColumns":1,"families":{"info":[{"timestamp":1434564411222,"tag":[],"qualifier":"","vlen":0}]},"row":"testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405."}, Delete{"ts":9223372036854775807,"totalColumns":1,"families":{"info":[{"timestamp":1434564411222,"tag":[],"qualifier":"","vlen":0}]},"row":"testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb."} 2015-06-17 11:06:51,228 INFO [ProcedureExecutorThread-5] hbase.MetaTableAccessor(1846): Deleted [{ENCODED => c555b4ca922c2de1fdd394fcc2ac20de, NAME => 'testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de.', STARTKEY => '', ENDKEY => 'a'}, {ENCODED => 3216ee6f55befe8693a09a4e6457d626, NAME => 'testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626.', STARTKEY => 'a', ENDKEY => 'b'}, {ENCODED => 7855514cb57ce135029b4fee0d052405, NAME => 'testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405.', STARTKEY => 'b', ENDKEY => 'c'}, {ENCODED => 3819afeab7512f52d645bda087c8d2eb, NAME => 'testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb.', STARTKEY => 'c', ENDKEY => ''}] 2015-06-17 11:06:51,307 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=6 2015-06-17 11:06:51,340 DEBUG [ProcedureExecutorThread-5] procedure.DeleteTableProcedure(126): delete 'testMasterProcsWithMasterRestarting' from filesystem 2015-06-17 11:06:51,343 DEBUG [ProcedureExecutorThread-5] procedure.DeleteTableProcedure(338): Archiving region testMasterProcsWithMasterRestarting,,1434564406676.c555b4ca922c2de1fdd394fcc2ac20de. from FS 2015-06-17 11:06:51,346 DEBUG [ProcedureExecutorThread-5] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/c555b4ca922c2de1fdd394fcc2ac20de 2015-06-17 11:06:51,352 DEBUG [ProcedureExecutorThread-5] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/c555b4ca922c2de1fdd394fcc2ac20de/f1, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/c555b4ca922c2de1fdd394fcc2ac20de/recovered.edits] 2015-06-17 11:06:51,364 DEBUG [ProcedureExecutorThread-5] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/c555b4ca922c2de1fdd394fcc2ac20de/recovered.edits/7.seqid, to hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/archive/data/default/testMasterProcsWithMasterRestarting/c555b4ca922c2de1fdd394fcc2ac20de/recovered.edits/7.seqid 2015-06-17 11:06:51,365 INFO [IPC Server handler 5 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741841_1017 127.0.0.1:50896 2015-06-17 11:06:51,365 DEBUG [ProcedureExecutorThread-5] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/c555b4ca922c2de1fdd394fcc2ac20de 2015-06-17 11:06:51,366 DEBUG [ProcedureExecutorThread-5] procedure.DeleteTableProcedure(338): Archiving region testMasterProcsWithMasterRestarting,a,1434564406676.3216ee6f55befe8693a09a4e6457d626. from FS 2015-06-17 11:06:51,366 DEBUG [ProcedureExecutorThread-5] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/3216ee6f55befe8693a09a4e6457d626 2015-06-17 11:06:51,369 DEBUG [ProcedureExecutorThread-5] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/3216ee6f55befe8693a09a4e6457d626/f1, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/3216ee6f55befe8693a09a4e6457d626/recovered.edits] 2015-06-17 11:06:51,379 DEBUG [ProcedureExecutorThread-5] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/3216ee6f55befe8693a09a4e6457d626/recovered.edits/7.seqid, to hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/archive/data/default/testMasterProcsWithMasterRestarting/3216ee6f55befe8693a09a4e6457d626/recovered.edits/7.seqid 2015-06-17 11:06:51,380 INFO [IPC Server handler 9 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741842_1018 127.0.0.1:50896 2015-06-17 11:06:51,380 DEBUG [ProcedureExecutorThread-5] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/3216ee6f55befe8693a09a4e6457d626 2015-06-17 11:06:51,380 DEBUG [ProcedureExecutorThread-5] procedure.DeleteTableProcedure(338): Archiving region testMasterProcsWithMasterRestarting,b,1434564406676.7855514cb57ce135029b4fee0d052405. from FS 2015-06-17 11:06:51,380 DEBUG [ProcedureExecutorThread-5] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/7855514cb57ce135029b4fee0d052405 2015-06-17 11:06:51,384 DEBUG [ProcedureExecutorThread-5] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/7855514cb57ce135029b4fee0d052405/f1, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/7855514cb57ce135029b4fee0d052405/recovered.edits] 2015-06-17 11:06:51,393 DEBUG [ProcedureExecutorThread-5] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/7855514cb57ce135029b4fee0d052405/recovered.edits/7.seqid, to hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/archive/data/default/testMasterProcsWithMasterRestarting/7855514cb57ce135029b4fee0d052405/recovered.edits/7.seqid 2015-06-17 11:06:51,394 INFO [IPC Server handler 8 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741843_1019 127.0.0.1:50896 2015-06-17 11:06:51,394 DEBUG [ProcedureExecutorThread-5] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/7855514cb57ce135029b4fee0d052405 2015-06-17 11:06:51,395 DEBUG [ProcedureExecutorThread-5] procedure.DeleteTableProcedure(338): Archiving region testMasterProcsWithMasterRestarting,c,1434564406676.3819afeab7512f52d645bda087c8d2eb. from FS 2015-06-17 11:06:51,395 DEBUG [ProcedureExecutorThread-5] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/3819afeab7512f52d645bda087c8d2eb 2015-06-17 11:06:51,397 DEBUG [ProcedureExecutorThread-5] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/3819afeab7512f52d645bda087c8d2eb/f1, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/3819afeab7512f52d645bda087c8d2eb/recovered.edits] 2015-06-17 11:06:51,408 DEBUG [ProcedureExecutorThread-5] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/3819afeab7512f52d645bda087c8d2eb/recovered.edits/7.seqid, to hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/archive/data/default/testMasterProcsWithMasterRestarting/3819afeab7512f52d645bda087c8d2eb/recovered.edits/7.seqid 2015-06-17 11:06:51,408 INFO [IPC Server handler 2 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741844_1020 127.0.0.1:50896 2015-06-17 11:06:51,409 DEBUG [ProcedureExecutorThread-5] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/3819afeab7512f52d645bda087c8d2eb 2015-06-17 11:06:51,409 DEBUG [ProcedureExecutorThread-5] procedure.DeleteTableProcedure(342): Table 'testMasterProcsWithMasterRestarting' archived! 2015-06-17 11:06:51,410 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=6 2015-06-17 11:06:51,410 INFO [IPC Server handler 8 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741836_1012 127.0.0.1:50896 2015-06-17 11:06:51,514 DEBUG [ProcedureExecutorThread-5] procedure.DeleteTableProcedure(132): delete 'testMasterProcsWithMasterRestarting' descriptor 2015-06-17 11:06:51,514 DEBUG [ProcedureExecutorThread-5] procedure.DeleteTableProcedure(400): Removing 'testMasterProcsWithMasterRestarting' descriptor. 2015-06-17 11:06:51,514 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=6 2015-06-17 11:06:51,617 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=6 2015-06-17 11:06:51,618 DEBUG [ProcedureExecutorThread-5] procedure.DeleteTableProcedure(137): delete 'testMasterProcsWithMasterRestarting' assignment state 2015-06-17 11:06:51,618 DEBUG [ProcedureExecutorThread-5] procedure.DeleteTableProcedure(390): Removing 'testMasterProcsWithMasterRestarting' from region states. 2015-06-17 11:06:51,619 DEBUG [ProcedureExecutorThread-5] procedure.DeleteTableProcedure(394): Marking 'testMasterProcsWithMasterRestarting' as deleted. 2015-06-17 11:06:51,619 DEBUG [ProcedureExecutorThread-5] hbase.MetaTableAccessor(1403): Delete{"ts":9223372036854775807,"totalColumns":1,"families":{"table":[{"timestamp":1434564411619,"tag":[],"qualifier":"state","vlen":0}]},"row":"testMasterProcsWithMasterRestarting"} 2015-06-17 11:06:51,625 INFO [ProcedureExecutorThread-5] hbase.MetaTableAccessor(1718): Deleted table testMasterProcsWithMasterRestarting state from META 2015-06-17 11:06:51,730 DEBUG [ProcedureExecutorThread-5] procedure.DeleteTableProcedure(143): delete 'testMasterProcsWithMasterRestarting' completed 2015-06-17 11:06:51,820 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=6 2015-06-17 11:06:51,835 DEBUG [ProcedureExecutorThread-5] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testMasterProcsWithMasterRestarting/write-master:509030000000004 2015-06-17 11:06:51,836 DEBUG [ProcedureExecutorThread-5] procedure2.ProcedureExecutor(729): Procedure completed in 740msec: DeleteTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=6 state=FINISHED 2015-06-17 11:06:52,024 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=6 2015-06-17 11:06:52,025 INFO [Thread-199] client.HBaseAdmin$TableFuture(4327): Operation: DELETE, Table Name: default:testMasterProcsWithMasterRestarting completed 2015-06-17 11:06:52,025 DEBUG [Thread-199] procedure.TestMasterRestartWithProcedures(153): executing step=0 took 5.368sec 2015-06-17 11:06:52,025 INFO [Thread-199] procedure.TestMasterRestartWithProcedures$1(104): executing create table step=1 2015-06-17 11:06:52,027 INFO [B.defaultRpcServer.handler=3,queue=0,port=50903] master.HMaster(1361): Client=ndimiduk/null create 'testMasterProcsWithMasterRestarting', {NAME => 'f1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2015-06-17 11:06:52,131 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50903] procedure2.ProcedureExecutor(566): Procedure CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=7 state=RUNNABLE added to the store. 2015-06-17 11:06:52,134 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:52,137 DEBUG [ProcedureExecutorThread-6] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testMasterProcsWithMasterRestarting/write-master:509030000000000 2015-06-17 11:06:52,137 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:52,141 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:52,146 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:52,155 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:52,169 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:52,193 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:52,236 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:52,255 INFO [IPC Server handler 0 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741845_1021{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 316 2015-06-17 11:06:52,338 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:52,441 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:52,544 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:52,647 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:52,663 DEBUG [ProcedureExecutorThread-6] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/data/default/testMasterProcsWithMasterRestarting/.tabledesc/.tableinfo.0000000001 2015-06-17 11:06:52,665 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:06:52,665 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:06:52,665 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:06:52,665 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:06:52,669 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=hregion-24877783.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-24877783, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/oldWALs 2015-06-17 11:06:52,669 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=hregion-85074294.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-85074294, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/oldWALs 2015-06-17 11:06:52,669 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=hregion-18881127.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-18881127, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/oldWALs 2015-06-17 11:06:52,669 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=hregion-36589148.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-36589148, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/oldWALs 2015-06-17 11:06:52,680 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] wal.FSHLog(1365): Slow sync cost: 7 ms, current pipeline: [] 2015-06-17 11:06:52,680 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-24877783/hregion-24877783.default.1434564412669 2015-06-17 11:06:52,681 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] wal.FSHLog(1365): Slow sync cost: 7 ms, current pipeline: [] 2015-06-17 11:06:52,681 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] wal.FSHLog(1365): Slow sync cost: 7 ms, current pipeline: [] 2015-06-17 11:06:52,681 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] wal.FSHLog(1365): Slow sync cost: 6 ms, current pipeline: [] 2015-06-17 11:06:52,681 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-18881127/hregion-18881127.default.1434564412669 2015-06-17 11:06:52,681 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-36589148/hregion-36589148.default.1434564412669 2015-06-17 11:06:52,681 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-85074294/hregion-85074294.default.1434564412669 2015-06-17 11:06:52,681 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] regionserver.HRegion(5888): creating HRegion testMasterProcsWithMasterRestarting HTD == 'testMasterProcsWithMasterRestarting', {NAME => 'f1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp Table name == testMasterProcsWithMasterRestarting 2015-06-17 11:06:52,682 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] regionserver.HRegion(5888): creating HRegion testMasterProcsWithMasterRestarting HTD == 'testMasterProcsWithMasterRestarting', {NAME => 'f1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp Table name == testMasterProcsWithMasterRestarting 2015-06-17 11:06:52,682 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] regionserver.HRegion(5888): creating HRegion testMasterProcsWithMasterRestarting HTD == 'testMasterProcsWithMasterRestarting', {NAME => 'f1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp Table name == testMasterProcsWithMasterRestarting 2015-06-17 11:06:52,682 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] regionserver.HRegion(5888): creating HRegion testMasterProcsWithMasterRestarting HTD == 'testMasterProcsWithMasterRestarting', {NAME => 'f1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp Table name == testMasterProcsWithMasterRestarting 2015-06-17 11:06:52,702 INFO [IPC Server handler 5 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741850_1026{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 72 2015-06-17 11:06:52,704 INFO [IPC Server handler 5 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741852_1028{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 72 2015-06-17 11:06:52,704 INFO [IPC Server handler 5 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741851_1027{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 71 2015-06-17 11:06:52,704 INFO [IPC Server handler 5 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741853_1029{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 71 2015-06-17 11:06:52,851 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:52,931 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(197): executing master kill 2015-06-17 11:06:52,931 INFO [MasterRestartThread] hbase.MiniHBaseCluster(422): Aborting 10.0.0.110,50903,1434564404489 2015-06-17 11:06:52,932 FATAL [MasterRestartThread] master.HMaster(1953): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2015-06-17 11:06:52,933 FATAL [MasterRestartThread] master.HMaster(1956): Aborting for tests java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:423) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:270) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$5.run(TestMasterRestartWithProcedures.java:198) 2015-06-17 11:06:52,935 INFO [MasterRestartThread] regionserver.HRegionServer(1859): STOPPED: Aborting for tests 2015-06-17 11:06:52,935 INFO [M:0;10.0.0.110:50903] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2015-06-17 11:06:52,935 INFO [MasterRestartThread] hbase.LocalHBaseCluster(379): Waiting on 10.0.0.110,50903,1434564404489 2015-06-17 11:06:52,935 INFO [SplitLogWorker-10.0.0.110:50903] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2015-06-17 11:06:52,935 INFO [M:0;10.0.0.110:50903] regionserver.HeapMemoryManager(201): Stoping HeapMemoryTuner chore. 2015-06-17 11:06:52,935 INFO [SplitLogWorker-10.0.0.110:50903] regionserver.SplitLogWorker(155): SplitLogWorker 10.0.0.110,50903,1434564404489 exiting 2015-06-17 11:06:52,936 INFO [M:0;10.0.0.110:50903] procedure2.ProcedureExecutor(490): Stopping the procedure executor 2015-06-17 11:06:52,936 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2015-06-17 11:06:52,936 INFO [M:0;10.0.0.110:50903] wal.WALProcedureStore(159): Stopping the WAL Procedure Store 2015-06-17 11:06:52,936 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2015-06-17 11:06:52,951 INFO [IPC Server handler 9 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741832_1008{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 14915 2015-06-17 11:06:52,985 INFO [10.0.0.110,50903,1434564404489_splitLogManager__ChoreService_1] hbase.ScheduledChore(179): Chore: SplitLogManager Timeout Monitor was stopped 2015-06-17 11:06:53,056 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:53,102 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] regionserver.HRegion(722): Instantiated testMasterProcsWithMasterRestarting,b,1434564412027.80aa2cbacbe6b871bf9e695f51afba04. 2015-06-17 11:06:53,102 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] regionserver.HRegion(722): Instantiated testMasterProcsWithMasterRestarting,c,1434564412027.f8d2e24ec38e2ce6edaed4a6b085790f. 2015-06-17 11:06:53,102 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] regionserver.HRegion(1365): Closing testMasterProcsWithMasterRestarting,b,1434564412027.80aa2cbacbe6b871bf9e695f51afba04.: disabling compactions & flushes 2015-06-17 11:06:53,102 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] regionserver.HRegion(722): Instantiated testMasterProcsWithMasterRestarting,a,1434564412027.6d23ad3fc2933eac8fe2002ed882feb0. 2015-06-17 11:06:53,102 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] regionserver.HRegion(1365): Closing testMasterProcsWithMasterRestarting,c,1434564412027.f8d2e24ec38e2ce6edaed4a6b085790f.: disabling compactions & flushes 2015-06-17 11:06:53,103 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] regionserver.HRegion(1365): Closing testMasterProcsWithMasterRestarting,a,1434564412027.6d23ad3fc2933eac8fe2002ed882feb0.: disabling compactions & flushes 2015-06-17 11:06:53,102 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] regionserver.HRegion(722): Instantiated testMasterProcsWithMasterRestarting,,1434564412027.ac712c0dba7f32a73cd3bc6eea75f38f. 2015-06-17 11:06:53,102 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] regionserver.HRegion(1392): Updates disabled for region testMasterProcsWithMasterRestarting,b,1434564412027.80aa2cbacbe6b871bf9e695f51afba04. 2015-06-17 11:06:53,103 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] regionserver.HRegion(1486): Closed testMasterProcsWithMasterRestarting,b,1434564412027.80aa2cbacbe6b871bf9e695f51afba04. 2015-06-17 11:06:53,103 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] regionserver.HRegion(1365): Closing testMasterProcsWithMasterRestarting,,1434564412027.ac712c0dba7f32a73cd3bc6eea75f38f.: disabling compactions & flushes 2015-06-17 11:06:53,103 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] regionserver.HRegion(1392): Updates disabled for region testMasterProcsWithMasterRestarting,a,1434564412027.6d23ad3fc2933eac8fe2002ed882feb0. 2015-06-17 11:06:53,103 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] regionserver.HRegion(1486): Closed testMasterProcsWithMasterRestarting,a,1434564412027.6d23ad3fc2933eac8fe2002ed882feb0. 2015-06-17 11:06:53,103 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] regionserver.HRegion(1392): Updates disabled for region testMasterProcsWithMasterRestarting,c,1434564412027.f8d2e24ec38e2ce6edaed4a6b085790f. 2015-06-17 11:06:53,103 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] regionserver.HRegion(1486): Closed testMasterProcsWithMasterRestarting,c,1434564412027.f8d2e24ec38e2ce6edaed4a6b085790f. 2015-06-17 11:06:53,103 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-36589148 2015-06-17 11:06:53,103 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] regionserver.HRegion(1392): Updates disabled for region testMasterProcsWithMasterRestarting,,1434564412027.ac712c0dba7f32a73cd3bc6eea75f38f. 2015-06-17 11:06:53,104 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] regionserver.HRegion(1486): Closed testMasterProcsWithMasterRestarting,,1434564412027.ac712c0dba7f32a73cd3bc6eea75f38f. 2015-06-17 11:06:53,104 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-85074294 2015-06-17 11:06:53,104 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-24877783 2015-06-17 11:06:53,104 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/WALs/hregion-18881127 2015-06-17 11:06:53,109 INFO [IPC Server handler 2 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741849_1025{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 83 2015-06-17 11:06:53,110 INFO [IPC Server handler 9 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741847_1023{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 91 2015-06-17 11:06:53,110 INFO [IPC Server handler 9 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741846_1022{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 91 2015-06-17 11:06:53,110 INFO [IPC Server handler 9 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741848_1024{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 91 2015-06-17 11:06:53,113 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/oldWALs 2015-06-17 11:06:53,113 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-3] wal.FSHLog(977): Closed WAL: FSHLog hregion-36589148.default:(num 1434564412669) 2015-06-17 11:06:53,258 INFO [10.0.0.110,50903,1434564404489_ChoreService_1] hbase.ScheduledChore(179): Chore: 10.0.0.110,50903,1434564404489-MemstoreFlusherChore was stopped 2015-06-17 11:06:53,261 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:53,265 INFO [master//10.0.0.110:0.logRoller] regionserver.LogRoller(159): LogRoller exiting. 2015-06-17 11:06:53,265 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(146): master//10.0.0.110:0.leaseChecker closing leases 2015-06-17 11:06:53,265 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(149): master//10.0.0.110:0.leaseChecker closed leases 2015-06-17 11:06:53,353 INFO [M:0;10.0.0.110:50903] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2015-06-17 11:06:53,353 INFO [M:0;10.0.0.110:50903] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2015-06-17 11:06:53,354 INFO [M:0;10.0.0.110:50903] regionserver.HRegionServer(1021): stopping server 10.0.0.110,50903,1434564404489 2015-06-17 11:06:53,354 DEBUG [M:0;10.0.0.110:50903] zookeeper.MetaTableLocator(617): Stopping MetaTableLocator 2015-06-17 11:06:53,354 DEBUG [RS_CLOSE_REGION-10.0.0.110:50903-0] handler.CloseRegionHandler(90): Processing close of hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:06:53,354 DEBUG [RS_CLOSE_REGION-10.0.0.110:50903-0] regionserver.HRegion(1365): Closing hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.: disabling compactions & flushes 2015-06-17 11:06:53,354 DEBUG [RS_CLOSE_REGION-10.0.0.110:50903-0] regionserver.HRegion(1392): Updates disabled for region hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:06:53,355 INFO [RS_CLOSE_REGION-10.0.0.110:50903-0] regionserver.HRegion(2111): Flushing 1/1 column families, memstore=344 B 2015-06-17 11:06:53,355 INFO [M:0;10.0.0.110:50903] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320003 2015-06-17 11:06:53,357 DEBUG [M:0;10.0.0.110:50903] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:06:53,358 INFO [M:0;10.0.0.110:50903] regionserver.CompactSplitThread(394): Waiting for Split Thread to finish... 2015-06-17 11:06:53,358 INFO [M:0;10.0.0.110:50903] regionserver.CompactSplitThread(394): Waiting for Merge Thread to finish... 2015-06-17 11:06:53,358 INFO [M:0;10.0.0.110:50903] regionserver.CompactSplitThread(394): Waiting for Large Compaction Thread to finish... 2015-06-17 11:06:53,358 INFO [M:0;10.0.0.110:50903] regionserver.CompactSplitThread(394): Waiting for Small Compaction Thread to finish... 2015-06-17 11:06:53,359 DEBUG [RpcServer.reader=2,bindAddress=10.0.0.110,port=50903] ipc.RpcServer$Listener(872): RpcServer.listener,port=50903: DISCONNECTING client 10.0.0.110:50923 because read count=-1. Number of active connections: 4 2015-06-17 11:06:53,359 INFO [M:0;10.0.0.110:50903] regionserver.HRegionServer(1243): Waiting on 2 regions to close 2015-06-17 11:06:53,359 DEBUG [M:0;10.0.0.110:50903] regionserver.HRegionServer(1247): {9ccc7b1e8942a5472c9cf2d844c2fff0=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., 1588230740=hbase:meta,,1.1588230740} 2015-06-17 11:06:53,359 DEBUG [RS_CLOSE_META-10.0.0.110:50903-0] handler.CloseRegionHandler(90): Processing close of hbase:meta,,1.1588230740 2015-06-17 11:06:53,359 DEBUG [RS_CLOSE_META-10.0.0.110:50903-0] regionserver.HRegion(1365): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2015-06-17 11:06:53,359 DEBUG [RS_CLOSE_META-10.0.0.110:50903-0] regionserver.HRegion(1392): Updates disabled for region hbase:meta,,1.1588230740 2015-06-17 11:06:53,360 INFO [RS_CLOSE_META-10.0.0.110:50903-0] regionserver.HRegion(2111): Flushing 2/2 column families, memstore=21.11 KB 2015-06-17 11:06:53,360 DEBUG [AsyncRpcChannel-pool2-t2] ipc.AsyncRpcChannel$8(574): IPC Client (184722895) connection to /10.0.0.110:50903 from ndimiduk: closed 2015-06-17 11:06:53,383 INFO [RS_OPEN_META-10.0.0.110:50903-0-MetaLogRoller] regionserver.LogRoller(159): LogRoller exiting. 2015-06-17 11:06:53,421 INFO [IPC Server handler 8 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741855_1031{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 6247 2015-06-17 11:06:53,422 INFO [IPC Server handler 8 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741854_1030{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 4912 2015-06-17 11:06:53,466 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:53,517 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/oldWALs 2015-06-17 11:06:53,517 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-1] wal.FSHLog(977): Closed WAL: FSHLog hregion-18881127.default:(num 1434564412669) 2015-06-17 11:06:53,517 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/oldWALs 2015-06-17 11:06:53,517 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-4] wal.FSHLog(977): Closed WAL: FSHLog hregion-85074294.default:(num 1434564412669) 2015-06-17 11:06:53,517 DEBUG [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/.tmp/oldWALs 2015-06-17 11:06:53,518 INFO [RegionOpenAndInitThread-testMasterProcsWithMasterRestarting-2] wal.FSHLog(977): Closed WAL: FSHLog hregion-24877783.default:(num 1434564412669) 2015-06-17 11:06:53,522 ERROR [ProcedureExecutorThread-6] server.NIOServerCnxnFactory$1(44): Thread Thread[ProcedureExecutorThread-6,5,main] died java.lang.AssertionError: recoverLease() must be called before inserting data at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.pushData(WALProcedureStore.java:422) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.update(WALProcedureStore.java:349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1009) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:713) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:666) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:70) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:453) 2015-06-17 11:06:53,673 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:53,824 INFO [RS_CLOSE_REGION-10.0.0.110:50903-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=8, memsize=344, hasBloomFilter=true, into tmp file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/.tmp/b88c94ff75ab4fafba9b040819600c6e 2015-06-17 11:06:53,824 INFO [RS_CLOSE_META-10.0.0.110:50903-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=101, memsize=19.0 K, hasBloomFilter=false, into tmp file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/fd7dd8c37dcd4f58adb7e614dd55c26b 2015-06-17 11:06:53,864 INFO [RS_CLOSE_META-10.0.0.110:50903-0] regionserver.StoreFile$Reader(1439): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fd7dd8c37dcd4f58adb7e614dd55c26b 2015-06-17 11:06:53,876 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:53,882 DEBUG [RS_CLOSE_REGION-10.0.0.110:50903-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/.tmp/b88c94ff75ab4fafba9b040819600c6e as hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/info/b88c94ff75ab4fafba9b040819600c6e 2015-06-17 11:06:53,889 INFO [IPC Server handler 7 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741856_1032{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 0 2015-06-17 11:06:53,890 INFO [RS_CLOSE_META-10.0.0.110:50903-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=101, memsize=2.1 K, hasBloomFilter=false, into tmp file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/adda056a1f284c47be8d8b15ee043a41 2015-06-17 11:06:53,892 INFO [RS_CLOSE_REGION-10.0.0.110:50903-0] regionserver.HStore(963): Added hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/info/b88c94ff75ab4fafba9b040819600c6e, entries=2, sequenceid=8, filesize=4.8 K 2015-06-17 11:06:53,893 INFO [RS_CLOSE_REGION-10.0.0.110:50903-0] regionserver.HRegion(2400): Finished memstore flush of ~344 B/344, currentsize=0 B/0 for region hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. in 539ms, sequenceid=8, compaction requested=false 2015-06-17 11:06:53,896 INFO [StoreCloserThread-hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.-1] regionserver.HStore(870): Closed info 2015-06-17 11:06:53,898 DEBUG [RS_CLOSE_META-10.0.0.110:50903-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/fd7dd8c37dcd4f58adb7e614dd55c26b as hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/fd7dd8c37dcd4f58adb7e614dd55c26b 2015-06-17 11:06:53,901 DEBUG [RS_CLOSE_REGION-10.0.0.110:50903-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/recovered.edits/11.seqid to file, newSeqId=11, maxSeqId=2 2015-06-17 11:06:53,902 INFO [RS_CLOSE_REGION-10.0.0.110:50903-0] regionserver.HRegion(1486): Closed hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:06:53,902 DEBUG [RS_CLOSE_REGION-10.0.0.110:50903-0] handler.CloseRegionHandler(122): Closed hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:06:53,905 INFO [RS_CLOSE_META-10.0.0.110:50903-0] regionserver.StoreFile$Reader(1439): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fd7dd8c37dcd4f58adb7e614dd55c26b 2015-06-17 11:06:53,906 INFO [RS_CLOSE_META-10.0.0.110:50903-0] regionserver.HStore(963): Added hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/fd7dd8c37dcd4f58adb7e614dd55c26b, entries=11, sequenceid=101, filesize=6.1 K 2015-06-17 11:06:53,907 DEBUG [RS_CLOSE_META-10.0.0.110:50903-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/adda056a1f284c47be8d8b15ee043a41 as hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/adda056a1f284c47be8d8b15ee043a41 2015-06-17 11:06:53,914 INFO [RS_CLOSE_META-10.0.0.110:50903-0] regionserver.HStore(963): Added hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/adda056a1f284c47be8d8b15ee043a41, entries=4, sequenceid=101, filesize=4.8 K 2015-06-17 11:06:53,916 INFO [RS_CLOSE_META-10.0.0.110:50903-0] regionserver.HRegion(2400): Finished memstore flush of ~21.11 KB/21616, currentsize=0 B/0 for region hbase:meta,,1.1588230740 in 556ms, sequenceid=101, compaction requested=false 2015-06-17 11:06:53,917 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(870): Closed info 2015-06-17 11:06:53,917 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(870): Closed table 2015-06-17 11:06:53,923 DEBUG [RS_CLOSE_META-10.0.0.110:50903-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/recovered.edits/104.seqid to file, newSeqId=104, maxSeqId=3 2015-06-17 11:06:53,923 DEBUG [RS_CLOSE_META-10.0.0.110:50903-0] coprocessor.CoprocessorHost(269): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2015-06-17 11:06:53,923 INFO [RS_CLOSE_META-10.0.0.110:50903-0] regionserver.HRegion(1486): Closed hbase:meta,,1.1588230740 2015-06-17 11:06:53,924 DEBUG [RS_CLOSE_META-10.0.0.110:50903-0] handler.CloseRegionHandler(122): Closed hbase:meta,,1.1588230740 2015-06-17 11:06:53,962 INFO [M:0;10.0.0.110:50903] regionserver.HRegionServer(1049): stopping server 10.0.0.110,50903,1434564404489; all regions closed. 2015-06-17 11:06:53,963 DEBUG [M:0;10.0.0.110:50903] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50903,1434564404489 2015-06-17 11:06:53,967 INFO [IPC Server handler 2 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741831_1007{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 13223 2015-06-17 11:06:54,080 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:54,285 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:54,373 DEBUG [M:0;10.0.0.110:50903] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:06:54,373 INFO [M:0;10.0.0.110:50903] wal.FSHLog(977): Closed WAL: FSHLog 10.0.0.110%2C50903%2C1434564404489..meta:.meta(num 1434564405379) 2015-06-17 11:06:54,373 DEBUG [M:0;10.0.0.110:50903] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50903,1434564404489 2015-06-17 11:06:54,375 INFO [IPC Server handler 1 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741829_1005{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 1381 2015-06-17 11:06:54,489 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:54,692 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50903] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:54,781 DEBUG [M:0;10.0.0.110:50903] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:06:54,781 INFO [M:0;10.0.0.110:50903] wal.FSHLog(977): Closed WAL: FSHLog 10.0.0.110%2C50903%2C1434564404489.default:(num 1434564405195) 2015-06-17 11:06:54,781 DEBUG [M:0;10.0.0.110:50903] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:06:54,781 INFO [M:0;10.0.0.110:50903] regionserver.Leases(146): M:0;10.0.0.110:50903 closing leases 2015-06-17 11:06:54,781 INFO [M:0;10.0.0.110:50903] regionserver.Leases(149): M:0;10.0.0.110:50903 closed leases 2015-06-17 11:06:54,781 INFO [M:0;10.0.0.110:50903] hbase.ChoreService(302): Chore service for: 10.0.0.110,50903,1434564404489 had [[ScheduledChore: Name: 10.0.0.110,50903,1434564404489-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-10.0.0.110:50903 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region 10.0.0.110,50903,1434564404489 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.0.0.110,50903,1434564404489-BalancerChore Period: 300000 Unit: MILLISECONDS]] on shutdown 2015-06-17 11:06:54,785 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/10.0.0.110,50903,1434564404489 2015-06-17 11:06:54,785 INFO [M:0;10.0.0.110:50903] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320007 2015-06-17 11:06:54,786 DEBUG [M:0;10.0.0.110:50903] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:06:54,786 DEBUG [M:0;10.0.0.110:50903] master.HMaster(1074): Stopping service threads 2015-06-17 11:06:54,787 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:06:54,787 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:06:54,787 INFO [M:0;10.0.0.110:50903] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320004 2015-06-17 11:06:54,788 DEBUG [main-EventThread] zookeeper.ZKUtil(488): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:06:54,788 DEBUG [main-EventThread] zookeeper.ZKUtil(488): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:06:54,788 DEBUG [M:0;10.0.0.110:50903] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:06:54,788 INFO [M:0;10.0.0.110:50903] hbase.ChoreService(302): Chore service for: 10.0.0.110,50903,1434564404489_splitLogManager_ had [] on shutdown 2015-06-17 11:06:54,788 DEBUG [AsyncRpcChannel-pool2-t5] ipc.AsyncRpcChannel$8(574): IPC Client (-662624958) connection to /10.0.0.110:50905 from ndimiduk: closed 2015-06-17 11:06:54,788 DEBUG [RpcServer.reader=1,bindAddress=10.0.0.110,port=50905] ipc.RpcServer$Listener(872): RpcServer.listener,port=50905: DISCONNECTING client 10.0.0.110:50940 because read count=-1. Number of active connections: 1 2015-06-17 11:06:54,788 INFO [M:0;10.0.0.110:50903] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2015-06-17 11:06:54,789 INFO [M:0;10.0.0.110:50903] ipc.RpcServer(2219): Stopping server on 50903 2015-06-17 11:06:54,789 INFO [RpcServer.listener,port=50903] ipc.RpcServer$Listener(776): RpcServer.listener,port=50903: stopping 2015-06-17 11:06:54,789 INFO [RpcServer.responder] ipc.RpcServer$Responder(1018): RpcServer.responder: stopped 2015-06-17 11:06:54,790 INFO [RpcServer.responder] ipc.RpcServer$Responder(921): RpcServer.responder: stopping 2015-06-17 11:06:54,791 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,50903,1434564404489 2015-06-17 11:06:54,791 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,50903,1434564404489 2015-06-17 11:06:54,791 INFO [main-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [10.0.0.110,50903,1434564404489] 2015-06-17 11:06:54,791 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): master:50903-0x14e02b3a2320000, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:06:54,792 INFO [M:0;10.0.0.110:50903] regionserver.HRegionServer(1093): stopping server 10.0.0.110,50903,1434564404489; zookeeper connection closed. 2015-06-17 11:06:54,792 INFO [M:0;10.0.0.110:50903] regionserver.HRegionServer(1096): M:0;10.0.0.110:50903 exiting 2015-06-17 11:06:54,792 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(200): executing master restart 2015-06-17 11:06:54,793 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:06:54,793 INFO [main-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(121): /hbase/rs/10.0.0.110,50903,1434564404489 znode expired, triggering replicatorRemoved event 2015-06-17 11:06:54,794 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:06:54,795 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:06:54,809 INFO [MasterRestartThread] client.ConnectionUtils(105): master//10.0.0.110:0 server-side HConnection retries=350 2015-06-17 11:06:54,809 INFO [MasterRestartThread] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2015-06-17 11:06:54,811 INFO [MasterRestartThread] ipc.RpcServer$Listener(598): master//10.0.0.110:0: started 10 reader(s). 2015-06-17 11:06:54,813 INFO [MasterRestartThread] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:06:54,814 INFO [MasterRestartThread] zookeeper.RecoverableZooKeeper(121): Process identifier=master:50957 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:06:54,816 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:509570x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:06:54,817 DEBUG [MasterRestartThread] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-17 11:06:54,818 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(578): master:50957-0x14e02b3a2320009 connected 2015-06-17 11:06:54,820 DEBUG [MasterRestartThread] zookeeper.ZKUtil(488): master:50957-0x14e02b3a2320009, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:06:54,820 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:50957-0x14e02b3a2320009, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2015-06-17 11:06:54,821 INFO [RpcServer.responder] ipc.RpcServer$Responder(917): RpcServer.responder: starting 2015-06-17 11:06:54,821 INFO [RpcServer.listener,port=50957] ipc.RpcServer$Listener(729): RpcServer.listener,port=50957: starting 2015-06-17 11:06:54,821 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=0 queue=0 2015-06-17 11:06:54,821 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=1 queue=0 2015-06-17 11:06:54,821 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=2 queue=0 2015-06-17 11:06:54,821 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=3 queue=0 2015-06-17 11:06:54,822 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=4 queue=0 2015-06-17 11:06:54,822 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=0 queue=0 2015-06-17 11:06:54,822 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=1 queue=1 2015-06-17 11:06:54,822 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=2 queue=0 2015-06-17 11:06:54,822 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=3 queue=1 2015-06-17 11:06:54,822 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=4 queue=0 2015-06-17 11:06:54,822 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=5 queue=1 2015-06-17 11:06:54,822 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=6 queue=0 2015-06-17 11:06:54,823 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=7 queue=1 2015-06-17 11:06:54,823 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=8 queue=0 2015-06-17 11:06:54,823 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=9 queue=1 2015-06-17 11:06:54,823 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=10 queue=0 2015-06-17 11:06:54,823 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=11 queue=1 2015-06-17 11:06:54,823 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=12 queue=0 2015-06-17 11:06:54,823 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=13 queue=1 2015-06-17 11:06:54,823 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=14 queue=0 2015-06-17 11:06:54,824 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=15 queue=1 2015-06-17 11:06:54,824 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=16 queue=0 2015-06-17 11:06:54,824 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=17 queue=1 2015-06-17 11:06:54,824 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=18 queue=0 2015-06-17 11:06:54,824 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=19 queue=1 2015-06-17 11:06:54,824 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=0 queue=0 2015-06-17 11:06:54,824 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=1 queue=0 2015-06-17 11:06:54,825 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=2 queue=0 2015-06-17 11:06:54,825 INFO [MasterRestartThread] master.HMaster(349): hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034, hbase.cluster.distributed=false 2015-06-17 11:06:54,825 INFO [MasterRestartThread] master.HMaster(1512): Adding backup master ZNode /hbase/backup-masters/10.0.0.110,50957,1434564414812 2015-06-17 11:06:54,826 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:50957-0x14e02b3a2320009, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/10.0.0.110,50957,1434564414812 2015-06-17 11:06:54,827 INFO [M:0;10.0.0.110:50957] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x6b42c5e6 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:06:54,827 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:50957-0x14e02b3a2320009, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:06:54,827 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:06:54,828 DEBUG [10.0.0.110:50957.activeMasterManager] zookeeper.ZKUtil(486): master:50957-0x14e02b3a2320009, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:06:54,829 INFO [10.0.0.110:50957.activeMasterManager] master.ActiveMasterManager(170): Deleting ZNode for /hbase/backup-masters/10.0.0.110,50957,1434564414812 from backup master directory 2015-06-17 11:06:54,829 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(486): master:50957-0x14e02b3a2320009, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:06:54,829 DEBUG [MasterRestartThread-EventThread] master.ActiveMasterManager(126): A master is now available 2015-06-17 11:06:54,829 DEBUG [M:0;10.0.0.110:50957-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x6b42c5e60x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:06:54,830 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:50957-0x14e02b3a2320009, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/10.0.0.110,50957,1434564414812 2015-06-17 11:06:54,830 WARN [10.0.0.110:50957.activeMasterManager] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:06:54,830 INFO [10.0.0.110:50957.activeMasterManager] master.ActiveMasterManager(179): Registered Active Master=10.0.0.110,50957,1434564414812 2015-06-17 11:06:54,830 DEBUG [M:0;10.0.0.110:50957] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6b95d222, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:06:54,830 DEBUG [M:0;10.0.0.110:50957] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:06:54,831 DEBUG [M:0;10.0.0.110:50957] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:06:54,831 DEBUG [M:0;10.0.0.110:50957-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x6b42c5e6-0x14e02b3a232000a connected 2015-06-17 11:06:54,833 INFO [M:0;10.0.0.110:50957] regionserver.HRegionServer(774): ClusterId : b7c4b21b-aab2-44b4-8562-2b61af22ad6c 2015-06-17 11:06:54,833 DEBUG [M:0;10.0.0.110:50957] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2015-06-17 11:06:54,834 DEBUG [M:0;10.0.0.110:50957] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:06:54,835 DEBUG [M:0;10.0.0.110:50957] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2015-06-17 11:06:54,835 DEBUG [M:0;10.0.0.110:50957] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2015-06-17 11:06:54,836 DEBUG [M:0;10.0.0.110:50957] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:06:54,837 DEBUG [M:0;10.0.0.110:50957] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2015-06-17 11:06:54,837 INFO [M:0;10.0.0.110:50957] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2015-06-17 11:06:54,838 INFO [M:0;10.0.0.110:50957] compactions.PressureAwareCompactionThroughputController(175): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2015-06-17 11:06:54,838 INFO [M:0;10.0.0.110:50957] regionserver.HRegionServer$CompactionChecker(1504): CompactionChecker runs every 1sec 2015-06-17 11:06:54,839 DEBUG [M:0;10.0.0.110:50957] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4839c2fb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=10.0.0.110/10.0.0.110:0 2015-06-17 11:06:54,839 DEBUG [M:0;10.0.0.110:50957] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:06:54,839 DEBUG [M:0;10.0.0.110:50957] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:06:54,839 DEBUG [M:0;10.0.0.110:50957] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:M:0;10.0.0.110:50957 2015-06-17 11:06:54,840 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:06:54,840 DEBUG [M:0;10.0.0.110:50957] zookeeper.ZKUtil(486): master:50957-0x14e02b3a2320009, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50957,1434564414812 2015-06-17 11:06:54,840 INFO [M:0;10.0.0.110:50957] regionserver.RegionServerCoprocessorHost(65): System coprocessor loading is enabled 2015-06-17 11:06:54,840 INFO [M:0;10.0.0.110:50957] regionserver.RegionServerCoprocessorHost(66): Table coprocessor loading is enabled 2015-06-17 11:06:54,841 INFO [M:0;10.0.0.110:50957] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,50957,1434564414812 with port=50957, startcode=1434564414812 2015-06-17 11:06:54,841 DEBUG [M:0;10.0.0.110:50957] regionserver.HRegionServer(2295): Master is not running yet 2015-06-17 11:06:54,841 WARN [M:0;10.0.0.110:50957] regionserver.HRegionServer(900): reportForDuty failed; sleeping and then retrying. 2015-06-17 11:06:54,841 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:06:54,841 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50957,1434564414812 2015-06-17 11:06:54,846 DEBUG [10.0.0.110:50957.activeMasterManager] util.FSTableDescriptors(807): Current tableInfoPath = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2015-06-17 11:06:54,849 DEBUG [10.0.0.110:50957.activeMasterManager] util.FSTableDescriptors(811): TableInfo already exists.. Skipping creation 2015-06-17 11:06:54,853 INFO [IPC Server handler 0 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741839_1015 127.0.0.1:50896 2015-06-17 11:06:54,853 INFO [IPC Server handler 0 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741838_1014 127.0.0.1:50896 2015-06-17 11:06:54,853 INFO [IPC Server handler 0 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741847_1023 127.0.0.1:50896 2015-06-17 11:06:54,853 INFO [IPC Server handler 0 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741846_1022 127.0.0.1:50896 2015-06-17 11:06:54,853 INFO [IPC Server handler 0 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741849_1025 127.0.0.1:50896 2015-06-17 11:06:54,854 INFO [IPC Server handler 0 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741837_1013 127.0.0.1:50896 2015-06-17 11:06:54,854 INFO [IPC Server handler 0 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741840_1016 127.0.0.1:50896 2015-06-17 11:06:54,854 INFO [IPC Server handler 0 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741834_1010 127.0.0.1:50896 2015-06-17 11:06:54,854 INFO [IPC Server handler 0 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741848_1024 127.0.0.1:50896 2015-06-17 11:06:54,855 INFO [10.0.0.110:50957.activeMasterManager] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:06:54,856 DEBUG [10.0.0.110:50957.activeMasterManager] coordination.ZKSplitLogManagerCoordination(885): Distributed log replay=true 2015-06-17 11:06:54,857 INFO [10.0.0.110:50957.activeMasterManager] coordination.ZKSplitLogManagerCoordination(599): Found 0 orphan tasks and 0 rescan nodes 2015-06-17 11:06:54,857 DEBUG [10.0.0.110:50957.activeMasterManager] util.FSTableDescriptors(222): Fetching table descriptors from the filesystem. 2015-06-17 11:06:54,868 INFO [10.0.0.110:50957.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x7d6d124c connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:06:54,870 DEBUG [10.0.0.110:50957.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x7d6d124c0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:06:54,871 DEBUG [10.0.0.110:50957.activeMasterManager] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7003cfd5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:06:54,871 DEBUG [10.0.0.110:50957.activeMasterManager] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:06:54,871 DEBUG [10.0.0.110:50957.activeMasterManager] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:06:54,872 DEBUG [10.0.0.110:50957.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x7d6d124c-0x14e02b3a232000b connected 2015-06-17 11:06:54,872 INFO [10.0.0.110:50957.activeMasterManager] balancer.StochasticLoadBalancer(135): loading config 2015-06-17 11:06:54,873 DEBUG [10.0.0.110:50957.activeMasterManager] zookeeper.ZKUtil(488): master:50957-0x14e02b3a2320009, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2015-06-17 11:06:54,874 DEBUG [10.0.0.110:50957.activeMasterManager] zookeeper.ZKUtil(486): master:50957-0x14e02b3a2320009, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:06:54,874 DEBUG [10.0.0.110:50957.activeMasterManager] zookeeper.ZKUtil(486): master:50957-0x14e02b3a2320009, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50957,1434564414812 2015-06-17 11:06:54,874 DEBUG [10.0.0.110:50957.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:06:54,875 DEBUG [10.0.0.110:50957.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,50957,1434564414812 2015-06-17 11:06:54,875 INFO [10.0.0.110:50957.activeMasterManager] master.HMaster(568): Server active/primary master=10.0.0.110,50957,1434564414812, sessionid=0x14e02b3a2320009, setting cluster-up flag (Was=true) 2015-06-17 11:06:54,876 DEBUG [10.0.0.110:50957.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:06:54,877 INFO [10.0.0.110:50957.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2015-06-17 11:06:54,878 DEBUG [10.0.0.110:50957.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,50957,1434564414812 2015-06-17 11:06:54,879 DEBUG [10.0.0.110:50957.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:06:54,879 INFO [10.0.0.110:50957.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2015-06-17 11:06:54,880 DEBUG [10.0.0.110:50957.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,50957,1434564414812 2015-06-17 11:06:54,880 INFO [10.0.0.110:50957.activeMasterManager] master.MasterCoprocessorHost(86): System coprocessor loading is enabled 2015-06-17 11:06:54,880 DEBUG [10.0.0.110:50957.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_OPEN_REGION-10.0.0.110:50957, corePoolSize=5, maxPoolSize=5 2015-06-17 11:06:54,881 DEBUG [10.0.0.110:50957.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_CLOSE_REGION-10.0.0.110:50957, corePoolSize=5, maxPoolSize=5 2015-06-17 11:06:54,881 DEBUG [10.0.0.110:50957.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_SERVER_OPERATIONS-10.0.0.110:50957, corePoolSize=5, maxPoolSize=5 2015-06-17 11:06:54,881 DEBUG [10.0.0.110:50957.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_META_SERVER_OPERATIONS-10.0.0.110:50957, corePoolSize=5, maxPoolSize=5 2015-06-17 11:06:54,881 DEBUG [10.0.0.110:50957.activeMasterManager] executor.ExecutorService(91): Starting executor service name=M_LOG_REPLAY_OPS-10.0.0.110:50957, corePoolSize=10, maxPoolSize=10 2015-06-17 11:06:54,881 DEBUG [10.0.0.110:50957.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_TABLE_OPERATIONS-10.0.0.110:50957, corePoolSize=1, maxPoolSize=1 2015-06-17 11:06:54,881 INFO [10.0.0.110:50957.activeMasterManager] procedure2.ProcedureExecutor(446): Starting procedure executor threads=9 2015-06-17 11:06:54,882 INFO [10.0.0.110:50957.activeMasterManager] wal.WALProcedureStore(214): Starting WAL Procedure Store lease recovery 2015-06-17 11:06:54,884 INFO [10.0.0.110:50957.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log 2015-06-17 11:06:54,888 INFO [10.0.0.110:50957.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log after 3ms 2015-06-17 11:06:54,888 DEBUG [10.0.0.110:50957.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log; isDirectory=false; length=14915; replication=1; blocksize=134217728; modification_time=1434564413352; access_time=1434564405122; owner=ndimiduk; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:06:54,893 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:06:54,895 DEBUG [AsyncRpcChannel-pool2-t6] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service MasterService, sasl=false 2015-06-17 11:06:54,896 DEBUG [RpcServer.listener,port=50957] ipc.RpcServer$Listener(840): RpcServer.listener,port=50957: connection from 10.0.0.110:50962; # active connections: 1 2015-06-17 11:06:54,897 INFO [RpcServer.reader=1,bindAddress=10.0.0.110,port=50957] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 50962 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:06:54,897 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50957: callId: 202 service: MasterService methodName: IsMasterRunning size: 24 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:54,898 DEBUG [10.0.0.110:50957.activeMasterManager] wal.WALProcedureStore(610): Roll new state log: 2 2015-06-17 11:06:54,900 INFO [10.0.0.110:50957.activeMasterManager] wal.WALProcedureStore(237): Lease acquired for flushLogId: 2 2015-06-17 11:06:54,903 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:06:54,905 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50957: callId: 204 service: MasterService methodName: IsMasterRunning size: 24 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:54,908 DEBUG [10.0.0.110:50957.activeMasterManager] procedure2.ProcedureExecutor$1(278): load procedures maxProcId=7 2015-06-17 11:06:54,909 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:06:54,910 DEBUG [10.0.0.110:50957.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=RUNNABLE isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=7 state=RUNNABLE 2015-06-17 11:06:54,911 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50957: callId: 206 service: MasterService methodName: IsMasterRunning size: 24 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:54,911 DEBUG [10.0.0.110:50957.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DeleteTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=6 state=FINISHED 2015-06-17 11:06:54,911 DEBUG [10.0.0.110:50957.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:06:54,911 DEBUG [10.0.0.110:50957.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=5 state=FINISHED 2015-06-17 11:06:54,911 DEBUG [10.0.0.110:50957.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:06:54,911 DEBUG [10.0.0.110:50957.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: EnableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=4 state=FINISHED 2015-06-17 11:06:54,911 DEBUG [10.0.0.110:50957.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:06:54,912 DEBUG [10.0.0.110:50957.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=3 state=FINISHED 2015-06-17 11:06:54,912 DEBUG [10.0.0.110:50957.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:06:54,912 DEBUG [10.0.0.110:50957.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=2 state=FINISHED 2015-06-17 11:06:54,912 DEBUG [10.0.0.110:50957.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:06:54,912 DEBUG [10.0.0.110:50957.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=hbase:namespace) user=ndimiduk (auth:SIMPLE) id=1 state=FINISHED 2015-06-17 11:06:54,912 DEBUG [10.0.0.110:50957.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:06:54,913 DEBUG [10.0.0.110:50957.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2015-06-17 11:06:54,914 INFO [10.0.0.110:50957.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:06:54,916 DEBUG [10.0.0.110:50957.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:06:54,916 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:06:54,916 DEBUG [10.0.0.110:50957.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/replication/rs already exists 2015-06-17 11:06:54,917 DEBUG [10.0.0.110:50957.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): replicationLogCleaner-0x14e02b3a232000c connected 2015-06-17 11:06:54,917 DEBUG [10.0.0.110:50957.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2015-06-17 11:06:54,918 DEBUG [10.0.0.110:50957.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner 2015-06-17 11:06:54,918 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50957: callId: 208 service: MasterService methodName: IsMasterRunning size: 24 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:54,918 DEBUG [10.0.0.110:50957.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2015-06-17 11:06:54,919 DEBUG [10.0.0.110:50957.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2015-06-17 11:06:54,919 DEBUG [10.0.0.110:50957.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2015-06-17 11:06:54,919 INFO [10.0.0.110:50957.activeMasterManager] master.ServerManager(1011): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 1, timeout of 4500 ms, interval of 1500 ms. 2015-06-17 11:06:54,919 INFO [M:0;10.0.0.110:50957] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,50957,1434564414812 with port=50957, startcode=1434564414812 2015-06-17 11:06:54,919 INFO [M:0;10.0.0.110:50957] master.ServerManager(459): Registering server=10.0.0.110,50957,1434564414812 2015-06-17 11:06:54,919 INFO [M:0;10.0.0.110:50957] regionserver.HRegionServer(1341): Config from master: hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034 2015-06-17 11:06:54,919 INFO [M:0;10.0.0.110:50957] regionserver.HRegionServer(1341): Config from master: fs.defaultFS=hdfs://localhost:50895 2015-06-17 11:06:54,919 INFO [M:0;10.0.0.110:50957] regionserver.HRegionServer(1341): Config from master: hbase.master.info.port=-1 2015-06-17 11:06:54,919 WARN [M:0;10.0.0.110:50957] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:06:54,919 INFO [M:0;10.0.0.110:50957] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:06:54,920 DEBUG [M:0;10.0.0.110:50957] regionserver.HRegionServer(1603): logdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50957,1434564414812 2015-06-17 11:06:54,923 DEBUG [M:0;10.0.0.110:50957] regionserver.Replication(142): ReplicationStatisticsThread 300 2015-06-17 11:06:54,923 INFO [M:0;10.0.0.110:50957] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:06:54,926 INFO [M:0;10.0.0.110:50957] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.0.0.110%2C50957%2C1434564414812.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50957,1434564414812, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:06:54,929 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:06:54,932 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:54,932 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50957: callId: 211 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:54,933 INFO [M:0;10.0.0.110:50957] wal.FSHLog(1365): Slow sync cost: 4 ms, current pipeline: [] 2015-06-17 11:06:54,933 INFO [M:0;10.0.0.110:50957] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50957,1434564414812/10.0.0.110%2C50957%2C1434564414812.default.1434564414926 2015-06-17 11:06:54,934 INFO [M:0;10.0.0.110:50957] regionserver.MetricsRegionServerWrapperImpl(113): Computing regionserver metrics every 5000 milliseconds 2015-06-17 11:06:54,935 DEBUG [M:0;10.0.0.110:50957] executor.ExecutorService(91): Starting executor service name=RS_OPEN_REGION-10.0.0.110:50957, corePoolSize=3, maxPoolSize=3 2015-06-17 11:06:54,935 DEBUG [M:0;10.0.0.110:50957] executor.ExecutorService(91): Starting executor service name=RS_OPEN_META-10.0.0.110:50957, corePoolSize=1, maxPoolSize=1 2015-06-17 11:06:54,935 DEBUG [M:0;10.0.0.110:50957] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_REGION-10.0.0.110:50957, corePoolSize=3, maxPoolSize=3 2015-06-17 11:06:54,935 DEBUG [M:0;10.0.0.110:50957] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_META-10.0.0.110:50957, corePoolSize=1, maxPoolSize=1 2015-06-17 11:06:54,935 DEBUG [M:0;10.0.0.110:50957] executor.ExecutorService(91): Starting executor service name=RS_LOG_REPLAY_OPS-10.0.0.110:50957, corePoolSize=2, maxPoolSize=2 2015-06-17 11:06:54,935 DEBUG [M:0;10.0.0.110:50957] executor.ExecutorService(91): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-10.0.0.110:50957, corePoolSize=3, maxPoolSize=3 2015-06-17 11:06:54,937 DEBUG [M:0;10.0.0.110:50957] zookeeper.ZKUtil(486): master:50957-0x14e02b3a2320009, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:06:54,937 DEBUG [M:0;10.0.0.110:50957] zookeeper.ZKUtil(486): master:50957-0x14e02b3a2320009, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50957,1434564414812 2015-06-17 11:06:54,937 INFO [M:0;10.0.0.110:50957] regionserver.ReplicationSourceManager(229): Current list of replicators: [10.0.0.110,50905,1434564404764, 10.0.0.110,50957,1434564414812] other RSs: [10.0.0.110,50905,1434564404764, 10.0.0.110,50957,1434564414812] 2015-06-17 11:06:54,952 INFO [M:0;10.0.0.110:50957] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x565ea8a connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:06:54,954 DEBUG [M:0;10.0.0.110:50957-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x565ea8a0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:06:54,954 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:54,954 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50957: callId: 213 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:54,955 DEBUG [M:0;10.0.0.110:50957] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodecWithTags@7ad84a86, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:06:54,955 DEBUG [M:0;10.0.0.110:50957] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:06:54,955 DEBUG [M:0;10.0.0.110:50957] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:06:54,955 DEBUG [M:0;10.0.0.110:50957-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x565ea8a-0x14e02b3a232000d connected 2015-06-17 11:06:54,969 INFO [M:0;10.0.0.110:50957] regionserver.HeapMemoryManager(190): Starting HeapMemoryTuner chore. 2015-06-17 11:06:54,969 INFO [SplitLogWorker-10.0.0.110:50957] regionserver.SplitLogWorker(134): SplitLogWorker 10.0.0.110,50957,1434564414812 starting 2015-06-17 11:06:54,969 INFO [M:0;10.0.0.110:50957] regionserver.HRegionServer(1363): Serving as 10.0.0.110,50957,1434564414812, RpcServer on 10.0.0.110/10.0.0.110:50957, sessionid=0x14e02b3a2320009 2015-06-17 11:06:54,969 INFO [10.0.0.110:50957.activeMasterManager] master.ServerManager(1028): Finished waiting for region servers count to settle; checked in 1, slept for 50 ms, expecting minimum of 1, maximum of 1, master is running 2015-06-17 11:06:54,969 DEBUG [M:0;10.0.0.110:50957] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2015-06-17 11:06:54,969 INFO [10.0.0.110:50957.activeMasterManager] master.ServerManager(459): Registering server=10.0.0.110,50905,1434564404764 2015-06-17 11:06:54,970 DEBUG [M:0;10.0.0.110:50957] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager 10.0.0.110,50957,1434564414812 2015-06-17 11:06:54,970 INFO [10.0.0.110:50957.activeMasterManager] master.HMaster(665): Registered server found up in zk but who has not yet reported in: 10.0.0.110,50905,1434564404764 2015-06-17 11:06:54,970 DEBUG [M:0;10.0.0.110:50957] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,50957,1434564414812' 2015-06-17 11:06:54,970 DEBUG [M:0;10.0.0.110:50957] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2015-06-17 11:06:54,970 DEBUG [M:0;10.0.0.110:50957] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2015-06-17 11:06:54,971 DEBUG [M:0;10.0.0.110:50957] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2015-06-17 11:06:54,971 DEBUG [M:0;10.0.0.110:50957] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2015-06-17 11:06:54,971 DEBUG [M:0;10.0.0.110:50957] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager 10.0.0.110,50957,1434564414812 2015-06-17 11:06:54,971 DEBUG [M:0;10.0.0.110:50957] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,50957,1434564414812' 2015-06-17 11:06:54,971 DEBUG [M:0;10.0.0.110:50957] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2015-06-17 11:06:54,971 DEBUG [M:0;10.0.0.110:50957] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2015-06-17 11:06:54,972 DEBUG [M:0;10.0.0.110:50957] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2015-06-17 11:06:54,972 INFO [M:0;10.0.0.110:50957] quotas.RegionServerQuotaManager(62): Quota support disabled 2015-06-17 11:06:54,973 INFO [10.0.0.110:50957.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50905,1434564404764 belongs to an existing region server 2015-06-17 11:06:54,973 INFO [10.0.0.110:50957.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50957,1434564414812 belongs to an existing region server 2015-06-17 11:06:54,975 WARN [10.0.0.110:50957.activeMasterManager] master.RegionStates(339): Open region's last host null should be the same as the current one 10.0.0.110,50903,1434564404489, ignored the last and used the current one 2015-06-17 11:06:54,996 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:54,996 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50957: callId: 215 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:55,098 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:55,098 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50957: callId: 217 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:55,200 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:55,200 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50957: callId: 219 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:55,302 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:55,302 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50957: callId: 221 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:55,404 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:55,404 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50957: callId: 223 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:55,405 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=512 ms ago, cancelled=false, msg= 2015-06-17 11:06:55,607 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:55,607 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50957: callId: 225 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:55,608 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=715 ms ago, cancelled=false, msg= 2015-06-17 11:06:55,809 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:55,809 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50957: callId: 227 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:55,809 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=916 ms ago, cancelled=false, msg= 2015-06-17 11:06:56,011 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:56,011 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50957: callId: 229 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:56,012 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1119 ms ago, cancelled=false, msg= 2015-06-17 11:06:56,213 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:56,213 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50957: callId: 231 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:56,214 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1321 ms ago, cancelled=false, msg= 2015-06-17 11:06:56,367 DEBUG [AsyncRpcChannel-pool2-t8] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2015-06-17 11:06:56,367 DEBUG [RpcServer.listener,port=50957] ipc.RpcServer$Listener(840): RpcServer.listener,port=50957: connection from 10.0.0.110:50967; # active connections: 2 2015-06-17 11:06:56,368 INFO [RpcServer.reader=2,bindAddress=10.0.0.110,port=50957] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 50967 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:06:56,416 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:56,416 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50957: callId: 233 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:56,417 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1524 ms ago, cancelled=false, msg= 2015-06-17 11:06:56,619 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:56,620 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50957: callId: 235 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:56,620 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1727 ms ago, cancelled=false, msg= 2015-06-17 11:06:56,822 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:56,823 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50957: callId: 237 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:56,823 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1930 ms ago, cancelled=false, msg= 2015-06-17 11:06:57,024 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:57,025 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50957: callId: 239 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:57,025 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2132 ms ago, cancelled=false, msg= 2015-06-17 11:06:57,227 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:57,227 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50957: callId: 241 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:57,228 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2335 ms ago, cancelled=false, msg= 2015-06-17 11:06:57,430 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:57,430 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50957: callId: 243 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:57,430 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2537 ms ago, cancelled=false, msg= 2015-06-17 11:06:57,632 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:57,632 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50957: callId: 245 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:57,633 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2740 ms ago, cancelled=false, msg= 2015-06-17 11:06:57,835 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:57,835 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50957: callId: 247 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:57,836 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2943 ms ago, cancelled=false, msg= 2015-06-17 11:06:58,039 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:58,039 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50957: callId: 249 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:58,041 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3148 ms ago, cancelled=false, msg= 2015-06-17 11:06:58,243 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:58,243 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50957: callId: 251 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:58,243 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3350 ms ago, cancelled=false, msg= 2015-06-17 11:06:58,446 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:58,446 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50957: callId: 253 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:58,447 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3554 ms ago, cancelled=false, msg= 2015-06-17 11:06:58,649 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:58,649 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50957: callId: 255 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:58,650 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3757 ms ago, cancelled=false, msg= 2015-06-17 11:06:58,852 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:58,852 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50957: callId: 257 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:58,853 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3960 ms ago, cancelled=false, msg= 2015-06-17 11:06:59,054 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:59,054 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50957: callId: 259 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:59,054 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4161 ms ago, cancelled=false, msg= 2015-06-17 11:06:59,258 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:59,258 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50957: callId: 261 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:59,258 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4365 ms ago, cancelled=false, msg= 2015-06-17 11:06:59,461 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:59,461 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50957: callId: 263 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:59,461 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4568 ms ago, cancelled=false, msg= 2015-06-17 11:06:59,664 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:59,665 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50957: callId: 265 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:59,665 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4772 ms ago, cancelled=false, msg= 2015-06-17 11:06:59,867 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:06:59,867 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50957: callId: 267 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:06:59,867 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4974 ms ago, cancelled=false, msg= 2015-06-17 11:07:00,069 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:00,070 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50957: callId: 269 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:00,070 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5177 ms ago, cancelled=false, msg= 2015-06-17 11:07:00,273 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:00,273 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50957: callId: 271 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:00,274 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5381 ms ago, cancelled=false, msg= 2015-06-17 11:07:00,275 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:06:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:06:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564414893, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:07:00,479 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:00,479 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50957: callId: 273 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:00,482 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:00,482 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50957: callId: 275 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:00,487 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:00,487 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50957: callId: 277 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:00,493 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:00,493 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50957: callId: 279 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:00,505 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:00,505 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50957: callId: 281 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:00,528 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:00,528 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50957: callId: 283 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:00,570 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:00,570 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50957: callId: 285 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:00,673 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:00,673 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50957: callId: 287 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:00,776 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:00,776 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50957: callId: 289 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:00,879 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:00,879 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50957: callId: 291 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:00,980 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:00,981 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50957: callId: 293 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:00,981 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=504 ms ago, cancelled=false, msg= 2015-06-17 11:07:01,183 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:01,184 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50957: callId: 295 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:01,184 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=707 ms ago, cancelled=false, msg= 2015-06-17 11:07:01,386 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:01,386 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50957: callId: 297 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:01,387 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=910 ms ago, cancelled=false, msg= 2015-06-17 11:07:01,589 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:01,589 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50957: callId: 299 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:01,590 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1113 ms ago, cancelled=false, msg= 2015-06-17 11:07:01,792 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:01,792 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50957: callId: 301 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:01,792 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1315 ms ago, cancelled=false, msg= 2015-06-17 11:07:01,995 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:01,995 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50957: callId: 303 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:01,995 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1518 ms ago, cancelled=false, msg= 2015-06-17 11:07:02,197 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:02,198 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50957: callId: 305 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:02,198 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1721 ms ago, cancelled=false, msg= 2015-06-17 11:07:02,399 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:02,399 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50957: callId: 307 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:02,400 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1923 ms ago, cancelled=false, msg= 2015-06-17 11:07:02,602 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:02,602 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50957: callId: 309 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:02,603 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2126 ms ago, cancelled=false, msg= 2015-06-17 11:07:02,805 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:02,805 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50957: callId: 311 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:02,806 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2329 ms ago, cancelled=false, msg= 2015-06-17 11:07:03,008 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:03,009 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50957: callId: 313 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:03,009 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2532 ms ago, cancelled=false, msg= 2015-06-17 11:07:03,211 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:03,211 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50957: callId: 315 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:03,212 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2735 ms ago, cancelled=false, msg= 2015-06-17 11:07:03,414 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:03,414 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50957: callId: 317 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:03,415 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2938 ms ago, cancelled=false, msg= 2015-06-17 11:07:03,617 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:03,617 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50957: callId: 319 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:03,618 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3141 ms ago, cancelled=false, msg= 2015-06-17 11:07:03,820 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:03,820 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50957: callId: 321 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:03,821 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3344 ms ago, cancelled=false, msg= 2015-06-17 11:07:04,022 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:04,022 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50957: callId: 323 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:04,023 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3546 ms ago, cancelled=false, msg= 2015-06-17 11:07:04,226 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:04,226 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50957: callId: 325 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:04,227 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3750 ms ago, cancelled=false, msg= 2015-06-17 11:07:04,428 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:04,428 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50957: callId: 327 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:04,428 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3951 ms ago, cancelled=false, msg= 2015-06-17 11:07:04,631 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:04,631 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50957: callId: 329 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:04,632 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4155 ms ago, cancelled=false, msg= 2015-06-17 11:07:04,834 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:04,834 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50957: callId: 331 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:04,835 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4358 ms ago, cancelled=false, msg= 2015-06-17 11:07:04,855 DEBUG [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(188): Sleeping for 4130 before executing next master restart count=1 2015-06-17 11:07:05,037 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:05,037 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50957: callId: 333 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:05,038 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4561 ms ago, cancelled=false, msg= 2015-06-17 11:07:05,241 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:05,241 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50957: callId: 335 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:05,242 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4765 ms ago, cancelled=false, msg= 2015-06-17 11:07:05,445 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:05,445 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50957: callId: 337 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:05,445 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4968 ms ago, cancelled=false, msg= 2015-06-17 11:07:05,648 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:05,648 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50957: callId: 339 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:05,649 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5172 ms ago, cancelled=false, msg= 2015-06-17 11:07:05,852 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:05,852 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50957: callId: 341 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:05,853 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5376 ms ago, cancelled=false, msg= 2015-06-17 11:07:05,854 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:07:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564420477, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:07:06,058 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:06,059 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50957: callId: 343 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:06,063 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:06,063 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50957: callId: 345 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:06,067 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:06,067 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50957: callId: 347 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:06,074 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:06,074 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50957: callId: 349 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:06,086 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:06,086 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50957: callId: 351 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:06,107 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:06,107 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50957: callId: 353 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:06,149 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:06,149 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50957: callId: 355 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:06,251 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:06,251 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50957: callId: 357 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:06,353 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:06,353 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50957: callId: 359 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:06,455 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:06,455 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50957: callId: 361 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:06,557 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:06,557 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50957: callId: 363 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:06,557 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=500 ms ago, cancelled=false, msg= 2015-06-17 11:07:06,759 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:06,759 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50957: callId: 365 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:06,760 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=703 ms ago, cancelled=false, msg= 2015-06-17 11:07:06,962 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:06,962 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50957: callId: 367 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:06,963 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=906 ms ago, cancelled=false, msg= 2015-06-17 11:07:07,166 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:07,166 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50957: callId: 369 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:07,167 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1110 ms ago, cancelled=false, msg= 2015-06-17 11:07:07,369 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:07,369 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50957: callId: 371 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:07,369 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1312 ms ago, cancelled=false, msg= 2015-06-17 11:07:07,572 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:07,572 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50957: callId: 373 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:07,572 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1515 ms ago, cancelled=false, msg= 2015-06-17 11:07:07,775 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:07,775 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50957: callId: 375 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:07,775 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1718 ms ago, cancelled=false, msg= 2015-06-17 11:07:07,978 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:07,978 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50957: callId: 377 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:07,979 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1922 ms ago, cancelled=false, msg= 2015-06-17 11:07:08,181 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:08,181 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50957: callId: 379 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:08,181 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2124 ms ago, cancelled=false, msg= 2015-06-17 11:07:08,386 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:08,386 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50957: callId: 381 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:08,386 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2329 ms ago, cancelled=false, msg= 2015-06-17 11:07:08,587 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:08,588 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50957: callId: 383 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:08,588 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2531 ms ago, cancelled=false, msg= 2015-06-17 11:07:08,790 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:08,791 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50957: callId: 385 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:08,791 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2734 ms ago, cancelled=false, msg= 2015-06-17 11:07:08,986 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(197): executing master kill 2015-06-17 11:07:08,986 INFO [MasterRestartThread] hbase.MiniHBaseCluster(422): Aborting 10.0.0.110,50957,1434564414812 2015-06-17 11:07:08,987 FATAL [MasterRestartThread] master.HMaster(1953): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2015-06-17 11:07:08,987 FATAL [MasterRestartThread] master.HMaster(1956): Aborting for tests java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:423) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:270) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$5.run(TestMasterRestartWithProcedures.java:198) 2015-06-17 11:07:08,987 INFO [MasterRestartThread] regionserver.HRegionServer(1859): STOPPED: Aborting for tests 2015-06-17 11:07:08,987 INFO [M:0;10.0.0.110:50957] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2015-06-17 11:07:08,987 INFO [MasterRestartThread] hbase.LocalHBaseCluster(379): Waiting on 10.0.0.110,50957,1434564414812 2015-06-17 11:07:08,987 INFO [M:0;10.0.0.110:50957] regionserver.HeapMemoryManager(201): Stoping HeapMemoryTuner chore. 2015-06-17 11:07:08,987 INFO [SplitLogWorker-10.0.0.110:50957] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2015-06-17 11:07:08,987 INFO [SplitLogWorker-10.0.0.110:50957] regionserver.SplitLogWorker(155): SplitLogWorker 10.0.0.110,50957,1434564414812 exiting 2015-06-17 11:07:08,988 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2015-06-17 11:07:08,987 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2015-06-17 11:07:08,987 INFO [M:0;10.0.0.110:50957] procedure2.ProcedureExecutor(490): Stopping the procedure executor 2015-06-17 11:07:08,988 INFO [M:0;10.0.0.110:50957] wal.WALProcedureStore(159): Stopping the WAL Procedure Store 2015-06-17 11:07:08,993 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:08,993 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50957: callId: 387 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:08,994 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2937 ms ago, cancelled=false, msg= 2015-06-17 11:07:08,995 INFO [IPC Server handler 9 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741858_1034{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 0 2015-06-17 11:07:08,995 INFO [M:0;10.0.0.110:50957] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2015-06-17 11:07:08,995 INFO [M:0;10.0.0.110:50957] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2015-06-17 11:07:08,996 INFO [M:0;10.0.0.110:50957] regionserver.HRegionServer(1021): stopping server 10.0.0.110,50957,1434564414812 2015-06-17 11:07:08,996 DEBUG [M:0;10.0.0.110:50957] zookeeper.MetaTableLocator(617): Stopping MetaTableLocator 2015-06-17 11:07:08,996 INFO [M:0;10.0.0.110:50957] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a232000a 2015-06-17 11:07:08,996 DEBUG [M:0;10.0.0.110:50957] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:07:08,997 INFO [M:0;10.0.0.110:50957] regionserver.HRegionServer(1049): stopping server 10.0.0.110,50957,1434564414812; all regions closed. 2015-06-17 11:07:08,997 DEBUG [M:0;10.0.0.110:50957] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50957,1434564414812 2015-06-17 11:07:08,997 INFO [10.0.0.110:50957.activeMasterManager] zookeeper.MetaTableLocator(340): Failed verification of hbase:meta,,1 at address=10.0.0.110,50903,1434564404489, exception=Call id=0 on server /10.0.0.110:50903 aborted: connection is closing 2015-06-17 11:07:08,997 DEBUG [AsyncRpcChannel-pool2-t7] ipc.AsyncRpcChannel$8(574): IPC Client (1622161989) connection to /10.0.0.110:50903 from ndimiduk.hfs.1: closed 2015-06-17 11:07:08,999 INFO [IPC Server handler 6 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741857_1033{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 83 2015-06-17 11:07:09,002 DEBUG [10.0.0.110:50957.activeMasterManager] zookeeper.ZKUtil(745): master:50957-0x14e02b3a2320009, quorum=localhost:51848, baseZNode=/hbase Unable to get data of znode /hbase/recovering-regions/1588230740 because node does not exist (not an error) 2015-06-17 11:07:09,002 DEBUG [M:0;10.0.0.110:50957] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:07:09,002 INFO [M:0;10.0.0.110:50957] wal.FSHLog(977): Closed WAL: FSHLog 10.0.0.110%2C50957%2C1434564414812.default:(num 1434564414926) 2015-06-17 11:07:09,002 DEBUG [M:0;10.0.0.110:50957] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:07:09,003 INFO [M:0;10.0.0.110:50957] regionserver.Leases(146): M:0;10.0.0.110:50957 closing leases 2015-06-17 11:07:09,003 INFO [M:0;10.0.0.110:50957] regionserver.Leases(149): M:0;10.0.0.110:50957 closed leases 2015-06-17 11:07:09,003 INFO [M:0;10.0.0.110:50957] hbase.ChoreService(302): Chore service for: 10.0.0.110,50957,1434564414812 had [[ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region 10.0.0.110,50957,1434564414812 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.0.0.110,50957,1434564414812-MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS]] on shutdown 2015-06-17 11:07:09,007 DEBUG [10.0.0.110:50957.activeMasterManager] coordination.ZKSplitLogManagerCoordination(650): Marked 1588230740 recovering from 10.0.0.110,50903,1434564404489: /hbase/recovering-regions/1588230740/10.0.0.110,50903,1434564404489 2015-06-17 11:07:09,008 INFO [10.0.0.110:50957.activeMasterManager] master.HMaster(884): Re-assigning hbase:meta with replicaId, 0 it was on 10.0.0.110,50903,1434564404489 2015-06-17 11:07:09,008 INFO [10.0.0.110:50957.activeMasterManager] master.RegionStates(1072): Transition {1588230740 state=OPEN, ts=1434564414975, server=10.0.0.110,50903,1434564404489} to {1588230740 state=OFFLINE, ts=1434564429008, server=10.0.0.110,50903,1434564404489} 2015-06-17 11:07:09,008 INFO [10.0.0.110:50957.activeMasterManager] zookeeper.MetaTableLocator(440): Setting hbase:meta region location in ZooKeeper as 10.0.0.110,50903,1434564404489 2015-06-17 11:07:09,008 INFO [10.0.0.110:50957.activeMasterManager] master.AssignmentManager(996): Skip assigning hbase:meta,,1.1588230740, the server is stopped/aborted 2015-06-17 11:07:09,008 FATAL [10.0.0.110:50957.activeMasterManager] master.HMaster$1(1546): Failed to become active master java.io.IOException: connection is closed at org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:254) at org.apache.hadoop.hbase.MetaTableAccessor.putToMetaTable(MetaTableAccessor.java:1341) at org.apache.hadoop.hbase.MetaTableAccessor.updateTableState(MetaTableAccessor.java:1691) at org.apache.hadoop.hbase.MetaTableAccessor.updateTableState(MetaTableAccessor.java:1110) at org.apache.hadoop.hbase.master.TableStateManager.udpateMetaState(TableStateManager.java:180) at org.apache.hadoop.hbase.master.TableStateManager.setTableState(TableStateManager.java:67) at org.apache.hadoop.hbase.master.HMaster.assignMeta(HMaster.java:892) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:711) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1542) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:09,199 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:09,199 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50957: callId: 389 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:09,200 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3143 ms ago, cancelled=false, msg= 2015-06-17 11:07:09,405 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:09,405 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50957: callId: 391 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:09,406 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3349 ms ago, cancelled=false, msg= 2015-06-17 11:07:09,612 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:09,612 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50957: callId: 393 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:09,613 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3556 ms ago, cancelled=false, msg= 2015-06-17 11:07:09,818 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:09,818 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50957] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50957: callId: 395 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50962 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:09,819 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3762 ms ago, cancelled=false, msg= 2015-06-17 11:07:09,858 INFO [10.0.0.110,50957,1434564414812_splitLogManager__ChoreService_1] hbase.ScheduledChore(179): Chore: SplitLogManager Timeout Monitor was stopped 2015-06-17 11:07:09,946 INFO [master//10.0.0.110:0.logRoller] regionserver.LogRoller(159): LogRoller exiting. 2015-06-17 11:07:09,946 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(146): master//10.0.0.110:0.leaseChecker closing leases 2015-06-17 11:07:09,947 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(149): master//10.0.0.110:0.leaseChecker closed leases 2015-06-17 11:07:09,947 INFO [M:0;10.0.0.110:50957] regionserver.CompactSplitThread(394): Waiting for Split Thread to finish... 2015-06-17 11:07:09,947 INFO [M:0;10.0.0.110:50957] regionserver.CompactSplitThread(394): Waiting for Merge Thread to finish... 2015-06-17 11:07:09,947 INFO [M:0;10.0.0.110:50957] regionserver.CompactSplitThread(394): Waiting for Large Compaction Thread to finish... 2015-06-17 11:07:09,947 INFO [M:0;10.0.0.110:50957] regionserver.CompactSplitThread(394): Waiting for Small Compaction Thread to finish... 2015-06-17 11:07:09,951 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:50957-0x14e02b3a2320009, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/10.0.0.110,50957,1434564414812 2015-06-17 11:07:09,951 INFO [M:0;10.0.0.110:50957] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a232000d 2015-06-17 11:07:09,953 DEBUG [M:0;10.0.0.110:50957] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:07:09,953 DEBUG [M:0;10.0.0.110:50957] master.HMaster(1074): Stopping service threads 2015-06-17 11:07:09,954 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:50957-0x14e02b3a2320009, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:07:09,955 INFO [M:0;10.0.0.110:50957] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a232000b 2015-06-17 11:07:09,955 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:07:09,955 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(488): master:50957-0x14e02b3a2320009, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:07:09,955 DEBUG [M:0;10.0.0.110:50957] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:07:09,955 DEBUG [main-EventThread] zookeeper.ZKUtil(488): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:07:09,956 INFO [M:0;10.0.0.110:50957] hbase.ChoreService(302): Chore service for: 10.0.0.110,50957,1434564414812_splitLogManager_ had [] on shutdown 2015-06-17 11:07:09,956 INFO [M:0;10.0.0.110:50957] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2015-06-17 11:07:09,956 INFO [M:0;10.0.0.110:50957] ipc.RpcServer(2219): Stopping server on 50957 2015-06-17 11:07:09,956 INFO [RpcServer.listener,port=50957] ipc.RpcServer$Listener(776): RpcServer.listener,port=50957: stopping 2015-06-17 11:07:09,957 INFO [RpcServer.responder] ipc.RpcServer$Responder(1018): RpcServer.responder: stopped 2015-06-17 11:07:09,957 INFO [RpcServer.responder] ipc.RpcServer$Responder(921): RpcServer.responder: stopping 2015-06-17 11:07:09,958 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:50957-0x14e02b3a2320009, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,50957,1434564414812 2015-06-17 11:07:09,958 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,50957,1434564414812 2015-06-17 11:07:09,958 INFO [MasterRestartThread-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [10.0.0.110,50957,1434564414812] 2015-06-17 11:07:09,958 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:50957-0x14e02b3a2320009, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:07:09,959 INFO [M:0;10.0.0.110:50957] regionserver.HRegionServer(1093): stopping server 10.0.0.110,50957,1434564414812; zookeeper connection closed. 2015-06-17 11:07:09,959 INFO [M:0;10.0.0.110:50957] regionserver.HRegionServer(1096): M:0;10.0.0.110:50957 exiting 2015-06-17 11:07:09,959 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(200): executing master restart 2015-06-17 11:07:09,959 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:09,959 INFO [main-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(121): /hbase/rs/10.0.0.110,50957,1434564414812 znode expired, triggering replicatorRemoved event 2015-06-17 11:07:09,959 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:07:09,960 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:09,981 INFO [MasterRestartThread] client.ConnectionUtils(105): master//10.0.0.110:0 server-side HConnection retries=350 2015-06-17 11:07:09,981 INFO [MasterRestartThread] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2015-06-17 11:07:09,983 INFO [MasterRestartThread] ipc.RpcServer$Listener(598): master//10.0.0.110:0: started 10 reader(s). 2015-06-17 11:07:09,986 INFO [MasterRestartThread] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:07:09,987 INFO [MasterRestartThread] zookeeper.RecoverableZooKeeper(121): Process identifier=master:50975 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:07:09,989 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:509750x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:07:09,990 DEBUG [MasterRestartThread] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-17 11:07:09,991 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(578): master:50975-0x14e02b3a232000e connected 2015-06-17 11:07:09,993 DEBUG [MasterRestartThread] zookeeper.ZKUtil(488): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:07:09,993 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2015-06-17 11:07:09,994 INFO [RpcServer.responder] ipc.RpcServer$Responder(917): RpcServer.responder: starting 2015-06-17 11:07:09,994 INFO [RpcServer.listener,port=50975] ipc.RpcServer$Listener(729): RpcServer.listener,port=50975: starting 2015-06-17 11:07:09,994 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=0 queue=0 2015-06-17 11:07:09,994 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=1 queue=0 2015-06-17 11:07:09,994 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=2 queue=0 2015-06-17 11:07:09,994 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=3 queue=0 2015-06-17 11:07:09,995 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=4 queue=0 2015-06-17 11:07:09,995 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=0 queue=0 2015-06-17 11:07:09,995 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=1 queue=1 2015-06-17 11:07:09,995 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=2 queue=0 2015-06-17 11:07:09,995 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=3 queue=1 2015-06-17 11:07:09,995 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=4 queue=0 2015-06-17 11:07:09,995 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=5 queue=1 2015-06-17 11:07:09,996 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=6 queue=0 2015-06-17 11:07:09,996 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=7 queue=1 2015-06-17 11:07:09,996 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=8 queue=0 2015-06-17 11:07:09,996 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=9 queue=1 2015-06-17 11:07:09,996 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=10 queue=0 2015-06-17 11:07:09,996 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=11 queue=1 2015-06-17 11:07:09,996 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=12 queue=0 2015-06-17 11:07:09,997 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=13 queue=1 2015-06-17 11:07:09,997 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=14 queue=0 2015-06-17 11:07:09,997 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=15 queue=1 2015-06-17 11:07:09,997 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=16 queue=0 2015-06-17 11:07:09,997 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=17 queue=1 2015-06-17 11:07:09,997 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=18 queue=0 2015-06-17 11:07:09,997 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=19 queue=1 2015-06-17 11:07:09,998 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=0 queue=0 2015-06-17 11:07:09,998 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=1 queue=0 2015-06-17 11:07:09,998 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=2 queue=0 2015-06-17 11:07:09,998 INFO [MasterRestartThread] master.HMaster(349): hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034, hbase.cluster.distributed=false 2015-06-17 11:07:09,998 INFO [MasterRestartThread] master.HMaster(1512): Adding backup master ZNode /hbase/backup-masters/10.0.0.110,50975,1434564429984 2015-06-17 11:07:09,999 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,000 INFO [M:0;10.0.0.110:50975] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x491e693a connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:07:10,000 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:07:10,000 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:07:10,001 DEBUG [10.0.0.110:50975.activeMasterManager] zookeeper.ZKUtil(486): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:07:10,001 INFO [10.0.0.110:50975.activeMasterManager] master.ActiveMasterManager(170): Deleting ZNode for /hbase/backup-masters/10.0.0.110,50975,1434564429984 from backup master directory 2015-06-17 11:07:10,002 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(486): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:07:10,002 DEBUG [MasterRestartThread-EventThread] master.ActiveMasterManager(126): A master is now available 2015-06-17 11:07:10,002 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,002 WARN [10.0.0.110:50975.activeMasterManager] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:07:10,002 INFO [10.0.0.110:50975.activeMasterManager] master.ActiveMasterManager(179): Registered Active Master=10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,003 DEBUG [M:0;10.0.0.110:50975-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x491e693a0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:07:10,004 DEBUG [M:0;10.0.0.110:50975] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5752db68, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:07:10,004 DEBUG [M:0;10.0.0.110:50975] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:07:10,004 DEBUG [M:0;10.0.0.110:50975] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:07:10,004 DEBUG [M:0;10.0.0.110:50975-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x491e693a-0x14e02b3a232000f connected 2015-06-17 11:07:10,005 INFO [M:0;10.0.0.110:50975] regionserver.HRegionServer(774): ClusterId : b7c4b21b-aab2-44b4-8562-2b61af22ad6c 2015-06-17 11:07:10,005 DEBUG [M:0;10.0.0.110:50975] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2015-06-17 11:07:10,006 DEBUG [M:0;10.0.0.110:50975] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:07:10,007 DEBUG [M:0;10.0.0.110:50975] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2015-06-17 11:07:10,007 DEBUG [M:0;10.0.0.110:50975] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2015-06-17 11:07:10,008 DEBUG [M:0;10.0.0.110:50975] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:07:10,008 DEBUG [M:0;10.0.0.110:50975] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2015-06-17 11:07:10,009 INFO [M:0;10.0.0.110:50975] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2015-06-17 11:07:10,009 INFO [M:0;10.0.0.110:50975] compactions.PressureAwareCompactionThroughputController(175): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2015-06-17 11:07:10,009 INFO [M:0;10.0.0.110:50975] regionserver.HRegionServer$CompactionChecker(1504): CompactionChecker runs every 1sec 2015-06-17 11:07:10,010 DEBUG [M:0;10.0.0.110:50975] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76daf0b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=10.0.0.110/10.0.0.110:0 2015-06-17 11:07:10,010 DEBUG [M:0;10.0.0.110:50975] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:07:10,010 DEBUG [M:0;10.0.0.110:50975] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:07:10,010 DEBUG [M:0;10.0.0.110:50975] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:M:0;10.0.0.110:50975 2015-06-17 11:07:10,011 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:07:10,011 DEBUG [M:0;10.0.0.110:50975] zookeeper.ZKUtil(486): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,011 INFO [M:0;10.0.0.110:50975] regionserver.RegionServerCoprocessorHost(65): System coprocessor loading is enabled 2015-06-17 11:07:10,011 INFO [M:0;10.0.0.110:50975] regionserver.RegionServerCoprocessorHost(66): Table coprocessor loading is enabled 2015-06-17 11:07:10,011 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:10,011 INFO [M:0;10.0.0.110:50975] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,50975,1434564429984 with port=50975, startcode=1434564429984 2015-06-17 11:07:10,012 DEBUG [M:0;10.0.0.110:50975] regionserver.HRegionServer(2295): Master is not running yet 2015-06-17 11:07:10,012 WARN [M:0;10.0.0.110:50975] regionserver.HRegionServer(900): reportForDuty failed; sleeping and then retrying. 2015-06-17 11:07:10,012 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,016 DEBUG [10.0.0.110:50975.activeMasterManager] util.FSTableDescriptors(807): Current tableInfoPath = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2015-06-17 11:07:10,019 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:07:10,019 DEBUG [10.0.0.110:50975.activeMasterManager] util.FSTableDescriptors(811): TableInfo already exists.. Skipping creation 2015-06-17 11:07:10,021 DEBUG [AsyncRpcChannel-pool2-t9] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service MasterService, sasl=false 2015-06-17 11:07:10,021 DEBUG [RpcServer.listener,port=50975] ipc.RpcServer$Listener(840): RpcServer.listener,port=50975: connection from 10.0.0.110:50980; # active connections: 1 2015-06-17 11:07:10,022 INFO [RpcServer.reader=1,bindAddress=10.0.0.110,port=50975] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 50980 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:07:10,022 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50975: callId: 397 service: MasterService methodName: IsMasterRunning size: 24 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:10,023 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3966 ms ago, cancelled=false, msg= 2015-06-17 11:07:10,024 INFO [10.0.0.110:50975.activeMasterManager] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:07:10,024 DEBUG [10.0.0.110:50975.activeMasterManager] coordination.ZKSplitLogManagerCoordination(885): Distributed log replay=true 2015-06-17 11:07:10,025 INFO [10.0.0.110:50975.activeMasterManager] coordination.ZKSplitLogManagerCoordination(599): Found 0 orphan tasks and 0 rescan nodes 2015-06-17 11:07:10,025 DEBUG [10.0.0.110:50975.activeMasterManager] util.FSTableDescriptors(222): Fetching table descriptors from the filesystem. 2015-06-17 11:07:10,039 INFO [10.0.0.110:50975.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x49f65429 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:07:10,041 DEBUG [10.0.0.110:50975.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x49f654290x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:07:10,041 DEBUG [10.0.0.110:50975.activeMasterManager] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56cdbcab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:07:10,041 DEBUG [10.0.0.110:50975.activeMasterManager] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:07:10,042 DEBUG [10.0.0.110:50975.activeMasterManager] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:07:10,042 DEBUG [10.0.0.110:50975.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x49f65429-0x14e02b3a2320010 connected 2015-06-17 11:07:10,043 INFO [10.0.0.110:50975.activeMasterManager] balancer.StochasticLoadBalancer(135): loading config 2015-06-17 11:07:10,043 DEBUG [10.0.0.110:50975.activeMasterManager] zookeeper.ZKUtil(488): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2015-06-17 11:07:10,044 DEBUG [10.0.0.110:50975.activeMasterManager] zookeeper.ZKUtil(486): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:10,044 DEBUG [10.0.0.110:50975.activeMasterManager] zookeeper.ZKUtil(486): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,045 DEBUG [10.0.0.110:50975.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:10,045 DEBUG [10.0.0.110:50975.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,045 INFO [10.0.0.110:50975.activeMasterManager] master.HMaster(568): Server active/primary master=10.0.0.110,50975,1434564429984, sessionid=0x14e02b3a232000e, setting cluster-up flag (Was=true) 2015-06-17 11:07:10,047 DEBUG [10.0.0.110:50975.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:07:10,047 INFO [10.0.0.110:50975.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2015-06-17 11:07:10,048 DEBUG [10.0.0.110:50975.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,049 DEBUG [10.0.0.110:50975.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:07:10,050 INFO [10.0.0.110:50975.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2015-06-17 11:07:10,050 DEBUG [10.0.0.110:50975.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,051 INFO [10.0.0.110:50975.activeMasterManager] master.MasterCoprocessorHost(86): System coprocessor loading is enabled 2015-06-17 11:07:10,051 DEBUG [10.0.0.110:50975.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_OPEN_REGION-10.0.0.110:50975, corePoolSize=5, maxPoolSize=5 2015-06-17 11:07:10,051 DEBUG [10.0.0.110:50975.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_CLOSE_REGION-10.0.0.110:50975, corePoolSize=5, maxPoolSize=5 2015-06-17 11:07:10,051 DEBUG [10.0.0.110:50975.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_SERVER_OPERATIONS-10.0.0.110:50975, corePoolSize=5, maxPoolSize=5 2015-06-17 11:07:10,051 DEBUG [10.0.0.110:50975.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_META_SERVER_OPERATIONS-10.0.0.110:50975, corePoolSize=5, maxPoolSize=5 2015-06-17 11:07:10,051 DEBUG [10.0.0.110:50975.activeMasterManager] executor.ExecutorService(91): Starting executor service name=M_LOG_REPLAY_OPS-10.0.0.110:50975, corePoolSize=10, maxPoolSize=10 2015-06-17 11:07:10,051 DEBUG [10.0.0.110:50975.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_TABLE_OPERATIONS-10.0.0.110:50975, corePoolSize=1, maxPoolSize=1 2015-06-17 11:07:10,052 INFO [10.0.0.110:50975.activeMasterManager] procedure2.ProcedureExecutor(446): Starting procedure executor threads=9 2015-06-17 11:07:10,052 INFO [10.0.0.110:50975.activeMasterManager] wal.WALProcedureStore(214): Starting WAL Procedure Store lease recovery 2015-06-17 11:07:10,053 INFO [10.0.0.110:50975.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log 2015-06-17 11:07:10,053 INFO [10.0.0.110:50975.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log after 0ms 2015-06-17 11:07:10,054 DEBUG [10.0.0.110:50975.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log; isDirectory=false; length=14915; replication=1; blocksize=134217728; modification_time=1434564413352; access_time=1434564405122; owner=ndimiduk; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:07:10,056 INFO [10.0.0.110:50975.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000002.log 2015-06-17 11:07:10,057 INFO [10.0.0.110:50975.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000002.log after 1ms 2015-06-17 11:07:10,057 DEBUG [10.0.0.110:50975.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000002.log; isDirectory=false; length=60; replication=1; blocksize=134217728; modification_time=1434564428995; access_time=1434564414897; owner=ndimiduk.hfs.1; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:07:10,062 DEBUG [10.0.0.110:50975.activeMasterManager] wal.WALProcedureStore(610): Roll new state log: 3 2015-06-17 11:07:10,063 INFO [10.0.0.110:50975.activeMasterManager] wal.WALProcedureStore(237): Lease acquired for flushLogId: 3 2015-06-17 11:07:10,064 INFO [10.0.0.110:50975.activeMasterManager] wal.ProcedureWALFormatReader(150): No active entry found in state log hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000002.log. removing it 2015-06-17 11:07:10,090 DEBUG [10.0.0.110:50975.activeMasterManager] procedure2.ProcedureExecutor$1(278): load procedures maxProcId=7 2015-06-17 11:07:10,092 DEBUG [10.0.0.110:50975.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=RUNNABLE isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=7 state=RUNNABLE 2015-06-17 11:07:10,092 DEBUG [10.0.0.110:50975.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DeleteTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=6 state=FINISHED 2015-06-17 11:07:10,092 DEBUG [10.0.0.110:50975.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:10,092 DEBUG [10.0.0.110:50975.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=5 state=FINISHED 2015-06-17 11:07:10,092 DEBUG [10.0.0.110:50975.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:10,092 DEBUG [10.0.0.110:50975.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: EnableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=4 state=FINISHED 2015-06-17 11:07:10,092 DEBUG [10.0.0.110:50975.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:10,093 DEBUG [10.0.0.110:50975.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=3 state=FINISHED 2015-06-17 11:07:10,093 DEBUG [10.0.0.110:50975.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:10,093 DEBUG [10.0.0.110:50975.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=2 state=FINISHED 2015-06-17 11:07:10,093 DEBUG [10.0.0.110:50975.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:10,093 DEBUG [10.0.0.110:50975.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=hbase:namespace) user=ndimiduk (auth:SIMPLE) id=1 state=FINISHED 2015-06-17 11:07:10,093 DEBUG [10.0.0.110:50975.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:10,093 DEBUG [10.0.0.110:50975.activeMasterManager] wal.WALProcedureStore(652): Remove log: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000002.log 2015-06-17 11:07:10,094 INFO [IPC Server handler 5 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741858_1034 127.0.0.1:50896 2015-06-17 11:07:10,096 DEBUG [10.0.0.110:50975.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2015-06-17 11:07:10,096 INFO [10.0.0.110:50975.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:07:10,098 DEBUG [10.0.0.110:50975.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:07:10,099 DEBUG [10.0.0.110:50975.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/replication/rs already exists 2015-06-17 11:07:10,099 DEBUG [10.0.0.110:50975.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2015-06-17 11:07:10,099 DEBUG [10.0.0.110:50975.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner 2015-06-17 11:07:10,100 DEBUG [10.0.0.110:50975.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): replicationLogCleaner-0x14e02b3a2320011 connected 2015-06-17 11:07:10,100 DEBUG [10.0.0.110:50975.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2015-06-17 11:07:10,100 DEBUG [10.0.0.110:50975.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2015-06-17 11:07:10,101 DEBUG [10.0.0.110:50975.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2015-06-17 11:07:10,101 INFO [10.0.0.110:50975.activeMasterManager] master.ServerManager(1011): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 1, timeout of 4500 ms, interval of 1500 ms. 2015-06-17 11:07:10,101 INFO [M:0;10.0.0.110:50975] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,50975,1434564429984 with port=50975, startcode=1434564429984 2015-06-17 11:07:10,101 INFO [M:0;10.0.0.110:50975] master.ServerManager(459): Registering server=10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,101 INFO [M:0;10.0.0.110:50975] regionserver.HRegionServer(1341): Config from master: hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034 2015-06-17 11:07:10,101 INFO [M:0;10.0.0.110:50975] regionserver.HRegionServer(1341): Config from master: fs.defaultFS=hdfs://localhost:50895 2015-06-17 11:07:10,101 INFO [M:0;10.0.0.110:50975] regionserver.HRegionServer(1341): Config from master: hbase.master.info.port=-1 2015-06-17 11:07:10,101 WARN [M:0;10.0.0.110:50975] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:07:10,101 INFO [M:0;10.0.0.110:50975] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:07:10,102 DEBUG [M:0;10.0.0.110:50975] regionserver.HRegionServer(1603): logdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,105 DEBUG [M:0;10.0.0.110:50975] regionserver.Replication(142): ReplicationStatisticsThread 300 2015-06-17 11:07:10,105 INFO [M:0;10.0.0.110:50975] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:07:10,107 INFO [M:0;10.0.0.110:50975] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.0.0.110%2C50975%2C1434564429984.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50975,1434564429984, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:07:10,112 INFO [M:0;10.0.0.110:50975] wal.FSHLog(1365): Slow sync cost: 3 ms, current pipeline: [] 2015-06-17 11:07:10,112 INFO [M:0;10.0.0.110:50975] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50975,1434564429984/10.0.0.110%2C50975%2C1434564429984.default.1434564430107 2015-06-17 11:07:10,113 INFO [M:0;10.0.0.110:50975] regionserver.MetricsRegionServerWrapperImpl(113): Computing regionserver metrics every 5000 milliseconds 2015-06-17 11:07:10,114 DEBUG [M:0;10.0.0.110:50975] executor.ExecutorService(91): Starting executor service name=RS_OPEN_REGION-10.0.0.110:50975, corePoolSize=3, maxPoolSize=3 2015-06-17 11:07:10,114 DEBUG [M:0;10.0.0.110:50975] executor.ExecutorService(91): Starting executor service name=RS_OPEN_META-10.0.0.110:50975, corePoolSize=1, maxPoolSize=1 2015-06-17 11:07:10,114 DEBUG [M:0;10.0.0.110:50975] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_REGION-10.0.0.110:50975, corePoolSize=3, maxPoolSize=3 2015-06-17 11:07:10,114 DEBUG [M:0;10.0.0.110:50975] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_META-10.0.0.110:50975, corePoolSize=1, maxPoolSize=1 2015-06-17 11:07:10,114 DEBUG [M:0;10.0.0.110:50975] executor.ExecutorService(91): Starting executor service name=RS_LOG_REPLAY_OPS-10.0.0.110:50975, corePoolSize=2, maxPoolSize=2 2015-06-17 11:07:10,114 DEBUG [M:0;10.0.0.110:50975] executor.ExecutorService(91): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-10.0.0.110:50975, corePoolSize=3, maxPoolSize=3 2015-06-17 11:07:10,116 DEBUG [M:0;10.0.0.110:50975] zookeeper.ZKUtil(486): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:10,116 DEBUG [M:0;10.0.0.110:50975] zookeeper.ZKUtil(486): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,116 INFO [M:0;10.0.0.110:50975] regionserver.ReplicationSourceManager(229): Current list of replicators: [10.0.0.110,50905,1434564404764, 10.0.0.110,50975,1434564429984] other RSs: [10.0.0.110,50905,1434564404764, 10.0.0.110,50975,1434564429984] 2015-06-17 11:07:10,130 INFO [M:0;10.0.0.110:50975] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x3311c90b connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:07:10,132 DEBUG [M:0;10.0.0.110:50975-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x3311c90b0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:07:10,133 DEBUG [M:0;10.0.0.110:50975] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodecWithTags@79af982, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:07:10,134 DEBUG [M:0;10.0.0.110:50975] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:07:10,134 DEBUG [M:0;10.0.0.110:50975] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:07:10,134 DEBUG [M:0;10.0.0.110:50975-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x3311c90b-0x14e02b3a2320012 connected 2015-06-17 11:07:10,147 INFO [M:0;10.0.0.110:50975] regionserver.HeapMemoryManager(190): Starting HeapMemoryTuner chore. 2015-06-17 11:07:10,147 INFO [SplitLogWorker-10.0.0.110:50975] regionserver.SplitLogWorker(134): SplitLogWorker 10.0.0.110,50975,1434564429984 starting 2015-06-17 11:07:10,147 INFO [M:0;10.0.0.110:50975] regionserver.HRegionServer(1363): Serving as 10.0.0.110,50975,1434564429984, RpcServer on 10.0.0.110/10.0.0.110:50975, sessionid=0x14e02b3a232000e 2015-06-17 11:07:10,147 DEBUG [M:0;10.0.0.110:50975] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2015-06-17 11:07:10,147 DEBUG [M:0;10.0.0.110:50975] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager 10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,148 DEBUG [M:0;10.0.0.110:50975] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,50975,1434564429984' 2015-06-17 11:07:10,148 DEBUG [M:0;10.0.0.110:50975] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2015-06-17 11:07:10,148 DEBUG [M:0;10.0.0.110:50975] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2015-06-17 11:07:10,148 DEBUG [M:0;10.0.0.110:50975] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2015-06-17 11:07:10,148 DEBUG [M:0;10.0.0.110:50975] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2015-06-17 11:07:10,148 DEBUG [M:0;10.0.0.110:50975] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager 10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,149 DEBUG [M:0;10.0.0.110:50975] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,50975,1434564429984' 2015-06-17 11:07:10,149 DEBUG [M:0;10.0.0.110:50975] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2015-06-17 11:07:10,149 DEBUG [M:0;10.0.0.110:50975] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2015-06-17 11:07:10,149 DEBUG [M:0;10.0.0.110:50975] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2015-06-17 11:07:10,149 INFO [M:0;10.0.0.110:50975] quotas.RegionServerQuotaManager(62): Quota support disabled 2015-06-17 11:07:10,151 INFO [10.0.0.110:50975.activeMasterManager] master.ServerManager(1028): Finished waiting for region servers count to settle; checked in 1, slept for 50 ms, expecting minimum of 1, maximum of 1, master is running 2015-06-17 11:07:10,151 INFO [10.0.0.110:50975.activeMasterManager] master.ServerManager(459): Registering server=10.0.0.110,50905,1434564404764 2015-06-17 11:07:10,151 INFO [10.0.0.110:50975.activeMasterManager] master.HMaster(665): Registered server found up in zk but who has not yet reported in: 10.0.0.110,50905,1434564404764 2015-06-17 11:07:10,154 INFO [10.0.0.110:50975.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50905,1434564404764 belongs to an existing region server 2015-06-17 11:07:10,155 INFO [10.0.0.110:50975.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50975,1434564429984 belongs to an existing region server 2015-06-17 11:07:10,157 DEBUG [10.0.0.110:50975.activeMasterManager] coordination.ZKSplitLogManagerCoordination(650): Marked 1588230740 recovering from 10.0.0.110,50903,1434564404489: /hbase/recovering-regions/1588230740/10.0.0.110,50903,1434564404489 2015-06-17 11:07:10,158 INFO [10.0.0.110:50975.activeMasterManager] master.HMaster(884): Re-assigning hbase:meta with replicaId, 0 it was on 10.0.0.110,50903,1434564404489 2015-06-17 11:07:10,158 DEBUG [10.0.0.110:50975.activeMasterManager] master.AssignmentManager(1239): No previous transition plan found (or ignoring an existing plan) for hbase:meta,,1.1588230740; generated random plan=hri=hbase:meta,,1.1588230740, src=, dest=10.0.0.110,50975,1434564429984; 2 (online=2) available servers, forceNewPlan=false 2015-06-17 11:07:10,158 INFO [10.0.0.110:50975.activeMasterManager] master.AssignmentManager(1028): Assigning hbase:meta,,1.1588230740 to 10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,158 INFO [10.0.0.110:50975.activeMasterManager] master.RegionStates(1072): Transition {1588230740 state=OFFLINE, ts=1434564430158, server=10.0.0.110,50903,1434564404489} to {1588230740 state=PENDING_OPEN, ts=1434564430158, server=10.0.0.110,50975,1434564429984} 2015-06-17 11:07:10,158 INFO [10.0.0.110:50975.activeMasterManager] zookeeper.MetaTableLocator(440): Setting hbase:meta region location in ZooKeeper as 10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,158 DEBUG [10.0.0.110:50975.activeMasterManager] master.ServerManager(937): New admin connection to 10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,159 INFO [10.0.0.110:50975.activeMasterManager] regionserver.RSRpcServices(1432): Open hbase:meta,,1.1588230740 2015-06-17 11:07:10,159 DEBUG [10.0.0.110:50975.activeMasterManager] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564430159,"tag":[],"qualifier":"state","vlen":2}]},"row":"hbase:meta"} 2015-06-17 11:07:10,159 INFO [RS_OPEN_META-10.0.0.110:50975-0] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:07:10,162 INFO [RS_OPEN_META-10.0.0.110:50975-0] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.0.0.110%2C50975%2C1434564429984..meta, suffix=.meta, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50975,1434564429984, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:07:10,168 INFO [RS_OPEN_META-10.0.0.110:50975-0] wal.FSHLog(1365): Slow sync cost: 3 ms, current pipeline: [] 2015-06-17 11:07:10,168 INFO [RS_OPEN_META-10.0.0.110:50975-0] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50975,1434564429984/10.0.0.110%2C50975%2C1434564429984..meta.1434564430162.meta 2015-06-17 11:07:10,168 DEBUG [RS_OPEN_META-10.0.0.110:50975-0] regionserver.HRegion(6070): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2015-06-17 11:07:10,170 DEBUG [RS_OPEN_META-10.0.0.110:50975-0] coprocessor.CoprocessorHost(179): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2015-06-17 11:07:10,170 DEBUG [RS_OPEN_META-10.0.0.110:50975-0] regionserver.HRegion(7284): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2015-06-17 11:07:10,171 INFO [RS_OPEN_META-10.0.0.110:50975-0] regionserver.RegionCoprocessorHost(350): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2015-06-17 11:07:10,171 DEBUG [RS_OPEN_META-10.0.0.110:50975-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table meta 1588230740 2015-06-17 11:07:10,171 DEBUG [RS_OPEN_META-10.0.0.110:50975-0] regionserver.HRegion(722): Instantiated hbase:meta,,1.1588230740 2015-06-17 11:07:10,175 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:07:10,175 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:07:10,186 INFO [StoreFileOpenerThread-info-1] regionserver.StoreFile$Reader(1439): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fd7dd8c37dcd4f58adb7e614dd55c26b 2015-06-17 11:07:10,186 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/fd7dd8c37dcd4f58adb7e614dd55c26b, isReference=false, isBulkLoadResult=false, seqid=101, majorCompaction=false 2015-06-17 11:07:10,187 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:07:10,188 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:07:10,196 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/adda056a1f284c47be8d8b15ee043a41, isReference=false, isBulkLoadResult=false, seqid=101, majorCompaction=false 2015-06-17 11:07:10,198 DEBUG [RS_OPEN_META-10.0.0.110:50975-0] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740 2015-06-17 11:07:10,201 DEBUG [RS_OPEN_META-10.0.0.110:50975-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:07:10,204 DEBUG [RS_OPEN_META-10.0.0.110:50975-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/recovered.edits/40000104.seqid to file, newSeqId=40000104, maxSeqId=104 2015-06-17 11:07:10,205 INFO [RS_OPEN_META-10.0.0.110:50975-0] regionserver.HRegion(848): Onlined 1588230740; next sequenceid=40000104 2015-06-17 11:07:10,205 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1893): Post open deploy tasks for hbase:meta,,1.1588230740 2015-06-17 11:07:10,207 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/recovering-regions/1588230740 2015-06-17 11:07:10,207 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(3181): Update last flushed sequence id of region 1588230740 for 10.0.0.110,50903,1434564404489 2015-06-17 11:07:10,208 DEBUG [PostOpenDeployTasks:1588230740] master.AssignmentManager(2812): Got transition OPENED for {1588230740 state=PENDING_OPEN, ts=1434564430158, server=10.0.0.110,50975,1434564429984} from 10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,208 INFO [PostOpenDeployTasks:1588230740] master.RegionStates(1072): Transition {1588230740 state=PENDING_OPEN, ts=1434564430158, server=10.0.0.110,50975,1434564429984} to {1588230740 state=OPEN, ts=1434564430208, server=10.0.0.110,50975,1434564429984} 2015-06-17 11:07:10,208 INFO [PostOpenDeployTasks:1588230740] zookeeper.MetaTableLocator(440): Setting hbase:meta region location in ZooKeeper as 10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,208 DEBUG [PostOpenDeployTasks:1588230740] master.RegionStates(417): Onlined 1588230740 on 10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,209 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1920): Finished post open deploy task for hbase:meta,,1.1588230740 2015-06-17 11:07:10,210 DEBUG [RS_OPEN_META-10.0.0.110:50975-0] handler.OpenRegionHandler(125): Opened hbase:meta,,1.1588230740 on 10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,223 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:07:10,225 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:10,225 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50975: callId: 400 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:10,226 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4169 ms ago, cancelled=false, msg= 2015-06-17 11:07:10,363 DEBUG [AsyncRpcChannel-pool2-t10] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service ClientService, sasl=false 2015-06-17 11:07:10,363 DEBUG [RpcServer.listener,port=50975] ipc.RpcServer$Listener(840): RpcServer.listener,port=50975: connection from 10.0.0.110:50987; # active connections: 2 2015-06-17 11:07:10,365 INFO [RpcServer.reader=2,bindAddress=10.0.0.110,port=50975] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 50987 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:07:10,368 INFO [10.0.0.110:50975.activeMasterManager] hbase.MetaTableAccessor(1692): Updated table hbase:meta state to ENABLED in META 2015-06-17 11:07:10,370 DEBUG [10.0.0.110:50975.activeMasterManager] master.MasterFileSystem(339): Renamed region directory: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50903,1434564404489-splitting 2015-06-17 11:07:10,370 INFO [10.0.0.110:50975.activeMasterManager] master.SplitLogManager(560): dead splitlog workers [10.0.0.110,50903,1434564404489] 2015-06-17 11:07:10,372 INFO [10.0.0.110:50975.activeMasterManager] master.SplitLogManager(192): hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50903,1434564404489-splitting is empty dir, no logs to split 2015-06-17 11:07:10,372 DEBUG [10.0.0.110:50975.activeMasterManager] master.SplitLogManager(255): Scheduling batch of logs to split 2015-06-17 11:07:10,372 INFO [10.0.0.110:50975.activeMasterManager] master.SplitLogManager(257): started splitting 0 logs in [hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50903,1434564404489-splitting] for [10.0.0.110,50903,1434564404489] 2015-06-17 11:07:10,374 DEBUG [10.0.0.110:50975.activeMasterManager] coordination.ZKSplitLogManagerCoordination(324): Processing recovering [1588230740] and servers [10.0.0.110,50903,1434564404489], isMetaRecovery=true 2015-06-17 11:07:10,377 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/recovering-regions/1588230740/10.0.0.110,50903,1434564404489 2015-06-17 11:07:10,379 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/recovering-regions/1588230740 2015-06-17 11:07:10,381 INFO [RS_OPEN_REGION-10.0.0.110:50975-0] handler.FinishRegionRecoveringHandler(52): /hbase/recovering-regions/1588230740 deleted; 1588230740 recovered. 2015-06-17 11:07:10,381 INFO [10.0.0.110:50975.activeMasterManager] master.SplitLogManager(316): finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50903,1434564404489-splitting] in 9ms 2015-06-17 11:07:10,381 DEBUG [10.0.0.110:50975.activeMasterManager] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564430381,"tag":[],"qualifier":"state","vlen":2}]},"row":"hbase:meta"} 2015-06-17 11:07:10,386 INFO [10.0.0.110:50975.activeMasterManager] hbase.MetaTableAccessor(1692): Updated table hbase:meta state to ENABLED in META 2015-06-17 11:07:10,399 WARN [10.0.0.110:50975.activeMasterManager] master.TableStateManager(216): Found table with no state, assuming ENABLED 2015-06-17 11:07:10,399 DEBUG [10.0.0.110:50975.activeMasterManager] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564430399,"tag":[],"qualifier":"state","vlen":2}]},"row":"testMasterProcsWithMasterRestarting"} 2015-06-17 11:07:10,401 INFO [10.0.0.110:50975.activeMasterManager] hbase.MetaTableAccessor(1692): Updated table testMasterProcsWithMasterRestarting state to ENABLED in META 2015-06-17 11:07:10,401 INFO [10.0.0.110:50975.activeMasterManager] master.ServerManager(686): AssignmentManager hasn't finished failover cleanup; waiting 2015-06-17 11:07:10,403 INFO [10.0.0.110:50975.activeMasterManager] master.HMaster(911): hbase:meta with replicaId 0 assigned=1, location=10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,426 INFO [10.0.0.110:50975.activeMasterManager] master.RegionStates(1072): Transition {9ccc7b1e8942a5472c9cf2d844c2fff0 state=OPEN, ts=1434564430420, server=10.0.0.110,50903,1434564404489} to {9ccc7b1e8942a5472c9cf2d844c2fff0 state=CLOSED, ts=1434564430426, server=10.0.0.110,50903,1434564404489} 2015-06-17 11:07:10,426 INFO [10.0.0.110:50975.activeMasterManager] master.AssignmentManager(538): Clean cluster startup. Assigning user regions 2015-06-17 11:07:10,426 INFO [10.0.0.110:50975.activeMasterManager] master.AssignmentManager(680): Assigning 1 region(s) to 10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,428 INFO [10.0.0.110:50975.activeMasterManager] master.RegionStates(1072): Transition {9ccc7b1e8942a5472c9cf2d844c2fff0 state=CLOSED, ts=1434564430426, server=10.0.0.110,50903,1434564404489} to {9ccc7b1e8942a5472c9cf2d844c2fff0 state=PENDING_OPEN, ts=1434564430428, server=10.0.0.110,50975,1434564429984} 2015-06-17 11:07:10,428 INFO [10.0.0.110:50975.activeMasterManager] master.RegionStateStore(207): Updating row hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. with state=PENDING_OPEN&sn=10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,428 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:10,428 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50975: callId: 402 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:10,429 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4372 ms ago, cancelled=false, msg= 2015-06-17 11:07:10,429 INFO [10.0.0.110:50975.activeMasterManager] regionserver.RSRpcServices(1432): Open hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:07:10,430 DEBUG [10.0.0.110:50975.activeMasterManager] master.AssignmentManager(851): Bulk assigning done for 10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,431 DEBUG [RS_OPEN_REGION-10.0.0.110:50975-1] regionserver.HRegion(6070): Opening region: {ENCODED => 9ccc7b1e8942a5472c9cf2d844c2fff0, NAME => 'hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.', STARTKEY => '', ENDKEY => ''} 2015-06-17 11:07:10,431 DEBUG [RS_OPEN_REGION-10.0.0.110:50975-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table namespace 9ccc7b1e8942a5472c9cf2d844c2fff0 2015-06-17 11:07:10,432 DEBUG [RS_OPEN_REGION-10.0.0.110:50975-1] regionserver.HRegion(722): Instantiated hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:07:10,434 INFO [StoreOpener-9ccc7b1e8942a5472c9cf2d844c2fff0-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=2, currentSize=1074496, freeSize=1042887808, maxSize=1043962304, heapSize=1074496, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:07:10,435 INFO [StoreOpener-9ccc7b1e8942a5472c9cf2d844c2fff0-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:07:10,440 INFO [10.0.0.110:50975.activeMasterManager] master.AssignmentManager(420): Joined the cluster in 37ms, failover=false 2015-06-17 11:07:10,444 DEBUG [10.0.0.110:50975.activeMasterManager] zookeeper.ZKUtil(486): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2015-06-17 11:07:10,447 DEBUG [10.0.0.110:50975.activeMasterManager] hbase.ZKNamespaceManager(196): Updating namespace cache from node default with data: \x0A\x07default 2015-06-17 11:07:10,447 DEBUG [10.0.0.110:50975.activeMasterManager] hbase.ZKNamespaceManager(196): Updating namespace cache from node hbase with data: \x0A\x05hbase 2015-06-17 11:07:10,448 DEBUG [StoreOpener-9ccc7b1e8942a5472c9cf2d844c2fff0-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/info/b88c94ff75ab4fafba9b040819600c6e, isReference=false, isBulkLoadResult=false, seqid=8, majorCompaction=false 2015-06-17 11:07:10,450 DEBUG [RS_OPEN_REGION-10.0.0.110:50975-1] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0 2015-06-17 11:07:10,452 DEBUG [RS_OPEN_REGION-10.0.0.110:50975-1] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:07:10,456 DEBUG [RS_OPEN_REGION-10.0.0.110:50975-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/recovered.edits/12.seqid to file, newSeqId=12, maxSeqId=11 2015-06-17 11:07:10,457 INFO [RS_OPEN_REGION-10.0.0.110:50975-1] regionserver.HRegion(848): Onlined 9ccc7b1e8942a5472c9cf2d844c2fff0; next sequenceid=12 2015-06-17 11:07:10,458 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] regionserver.HRegionServer(1893): Post open deploy tasks for hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:07:10,459 DEBUG [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.AssignmentManager(2812): Got transition OPENED for {9ccc7b1e8942a5472c9cf2d844c2fff0 state=PENDING_OPEN, ts=1434564430428, server=10.0.0.110,50975,1434564429984} from 10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,459 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStates(1072): Transition {9ccc7b1e8942a5472c9cf2d844c2fff0 state=PENDING_OPEN, ts=1434564430428, server=10.0.0.110,50975,1434564429984} to {9ccc7b1e8942a5472c9cf2d844c2fff0 state=OPEN, ts=1434564430459, server=10.0.0.110,50975,1434564429984} 2015-06-17 11:07:10,459 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStateStore(207): Updating row hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. with state=OPEN&openSeqNum=12&server=10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,460 DEBUG [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStates(417): Onlined 9ccc7b1e8942a5472c9cf2d844c2fff0 on 10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,460 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStates(429): Offlined 9ccc7b1e8942a5472c9cf2d844c2fff0 from 10.0.0.110,50903,1434564404489 2015-06-17 11:07:10,461 DEBUG [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] regionserver.HRegionServer(1920): Finished post open deploy task for hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:07:10,462 DEBUG [RS_OPEN_REGION-10.0.0.110:50975-1] handler.OpenRegionHandler(125): Opened hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. on 10.0.0.110,50975,1434564429984 2015-06-17 11:07:10,630 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:10,630 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50975: callId: 404 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:10,631 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4574 ms ago, cancelled=false, msg= 2015-06-17 11:07:10,833 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:10,833 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50975: callId: 406 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:10,834 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4777 ms ago, cancelled=false, msg= 2015-06-17 11:07:11,035 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:11,035 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50975: callId: 408 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:11,035 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4978 ms ago, cancelled=false, msg= 2015-06-17 11:07:11,238 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:11,238 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50975: callId: 410 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:11,239 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5182 ms ago, cancelled=false, msg= 2015-06-17 11:07:11,391 DEBUG [AsyncRpcChannel-pool2-t12] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2015-06-17 11:07:11,392 DEBUG [RpcServer.listener,port=50975] ipc.RpcServer$Listener(840): RpcServer.listener,port=50975: connection from 10.0.0.110:50990; # active connections: 3 2015-06-17 11:07:11,392 INFO [RpcServer.reader=3,bindAddress=10.0.0.110,port=50975] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 50990 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:07:11,441 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:11,441 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50975: callId: 412 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:11,442 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5385 ms ago, cancelled=false, msg= 2015-06-17 11:07:11,443 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:07:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564426057, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:07:11,646 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:11,646 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50975: callId: 414 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:11,649 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:11,649 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50975: callId: 416 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:11,654 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:11,654 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50975: callId: 418 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:11,660 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:11,660 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50975: callId: 420 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:11,671 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:11,671 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50975: callId: 422 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:11,693 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:11,693 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50975: callId: 424 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:11,735 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:11,735 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50975: callId: 426 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:11,837 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:11,837 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50975: callId: 428 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:11,939 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:11,939 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50975: callId: 430 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:12,040 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:12,040 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50975: callId: 432 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:12,142 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:12,142 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50975: callId: 434 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:12,143 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=498 ms ago, cancelled=false, msg= 2015-06-17 11:07:12,345 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:12,345 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50975: callId: 436 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:12,346 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=701 ms ago, cancelled=false, msg= 2015-06-17 11:07:12,547 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:12,547 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50975: callId: 438 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:12,547 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=902 ms ago, cancelled=false, msg= 2015-06-17 11:07:12,750 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:12,750 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50975: callId: 440 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:12,751 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1106 ms ago, cancelled=false, msg= 2015-06-17 11:07:12,953 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:12,953 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50975: callId: 442 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:12,953 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1308 ms ago, cancelled=false, msg= 2015-06-17 11:07:13,157 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:13,157 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50975: callId: 444 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:13,157 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1512 ms ago, cancelled=false, msg= 2015-06-17 11:07:13,359 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:13,359 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50975: callId: 446 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:13,359 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1714 ms ago, cancelled=false, msg= 2015-06-17 11:07:13,561 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:13,562 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50975: callId: 448 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:13,562 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1917 ms ago, cancelled=false, msg= 2015-06-17 11:07:13,764 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:13,764 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50975: callId: 450 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:13,765 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2120 ms ago, cancelled=false, msg= 2015-06-17 11:07:13,967 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:13,967 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50975: callId: 452 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:13,968 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2323 ms ago, cancelled=false, msg= 2015-06-17 11:07:14,171 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:14,171 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50975: callId: 454 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:14,171 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2526 ms ago, cancelled=false, msg= 2015-06-17 11:07:14,372 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:14,373 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50975: callId: 456 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:14,373 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2728 ms ago, cancelled=false, msg= 2015-06-17 11:07:14,574 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:14,575 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50975: callId: 458 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:14,575 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2930 ms ago, cancelled=false, msg= 2015-06-17 11:07:14,776 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:14,776 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50975: callId: 460 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:14,777 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3132 ms ago, cancelled=false, msg= 2015-06-17 11:07:14,980 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:14,980 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50975: callId: 462 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:14,980 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3335 ms ago, cancelled=false, msg= 2015-06-17 11:07:15,183 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:15,184 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50975: callId: 464 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:15,184 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3539 ms ago, cancelled=false, msg= 2015-06-17 11:07:15,387 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:15,387 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50975: callId: 466 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:15,387 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3742 ms ago, cancelled=false, msg= 2015-06-17 11:07:15,590 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:15,591 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50975: callId: 468 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:15,591 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3946 ms ago, cancelled=false, msg= 2015-06-17 11:07:15,793 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:15,793 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50975: callId: 470 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:15,794 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4149 ms ago, cancelled=false, msg= 2015-06-17 11:07:15,996 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:15,996 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50975: callId: 472 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:15,996 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4351 ms ago, cancelled=false, msg= 2015-06-17 11:07:16,199 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:16,199 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50975: callId: 474 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:16,199 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4554 ms ago, cancelled=false, msg= 2015-06-17 11:07:16,401 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:16,401 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50975: callId: 476 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:16,402 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4757 ms ago, cancelled=false, msg= 2015-06-17 11:07:16,604 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:16,604 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50975: callId: 478 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:16,605 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4960 ms ago, cancelled=false, msg= 2015-06-17 11:07:16,807 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:16,807 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50975: callId: 480 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:16,807 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5162 ms ago, cancelled=false, msg= 2015-06-17 11:07:17,010 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:17,010 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50975: callId: 482 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:17,010 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5365 ms ago, cancelled=false, msg= 2015-06-17 11:07:17,011 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:07:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564431645, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:07:17,215 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:17,215 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50975: callId: 484 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:17,218 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:17,218 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50975: callId: 486 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:17,223 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:17,223 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50975: callId: 488 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:17,230 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:17,230 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50975: callId: 490 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:17,241 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:17,241 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50975: callId: 492 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:17,262 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:17,263 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50975: callId: 494 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:17,305 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:17,305 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50975: callId: 496 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:17,406 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:17,406 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50975: callId: 498 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:17,508 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:17,508 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50975: callId: 500 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:17,610 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:17,611 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50975: callId: 502 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:17,712 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:17,712 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50975: callId: 504 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:17,713 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=499 ms ago, cancelled=false, msg= 2015-06-17 11:07:17,915 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:17,915 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50975: callId: 506 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:17,915 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=701 ms ago, cancelled=false, msg= 2015-06-17 11:07:18,117 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:18,117 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50975: callId: 508 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:18,117 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=903 ms ago, cancelled=false, msg= 2015-06-17 11:07:18,320 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:18,320 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50975: callId: 510 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:18,321 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1107 ms ago, cancelled=false, msg= 2015-06-17 11:07:18,523 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:18,523 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50975: callId: 512 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:18,523 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1309 ms ago, cancelled=false, msg= 2015-06-17 11:07:18,727 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:18,727 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50975: callId: 514 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:18,727 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1513 ms ago, cancelled=false, msg= 2015-06-17 11:07:18,929 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:18,929 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50975: callId: 516 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:18,930 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1716 ms ago, cancelled=false, msg= 2015-06-17 11:07:19,132 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:19,132 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50975: callId: 518 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:19,133 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1919 ms ago, cancelled=false, msg= 2015-06-17 11:07:19,335 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:19,335 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50975: callId: 520 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:19,336 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2122 ms ago, cancelled=false, msg= 2015-06-17 11:07:19,538 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:19,539 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50975: callId: 522 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:19,539 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2325 ms ago, cancelled=false, msg= 2015-06-17 11:07:19,741 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:19,741 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50975: callId: 524 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:19,741 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2527 ms ago, cancelled=false, msg= 2015-06-17 11:07:19,944 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:19,944 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50975: callId: 526 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:19,945 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2731 ms ago, cancelled=false, msg= 2015-06-17 11:07:20,040 DEBUG [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(188): Sleeping for 5660 before executing next master restart count=2 2015-06-17 11:07:20,148 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:20,148 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50975: callId: 528 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:20,149 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2935 ms ago, cancelled=false, msg= 2015-06-17 11:07:20,351 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:20,351 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50975: callId: 530 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:20,352 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3138 ms ago, cancelled=false, msg= 2015-06-17 11:07:20,554 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:20,554 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50975: callId: 532 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:20,554 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3340 ms ago, cancelled=false, msg= 2015-06-17 11:07:20,755 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:20,755 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50975: callId: 534 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:20,756 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3542 ms ago, cancelled=false, msg= 2015-06-17 11:07:20,959 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:20,959 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50975: callId: 536 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:20,959 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3745 ms ago, cancelled=false, msg= 2015-06-17 11:07:21,161 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:21,161 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50975: callId: 538 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:21,161 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3947 ms ago, cancelled=false, msg= 2015-06-17 11:07:21,363 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:21,363 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50975: callId: 540 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:21,363 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4149 ms ago, cancelled=false, msg= 2015-06-17 11:07:21,565 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:21,566 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50975: callId: 542 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:21,566 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4352 ms ago, cancelled=false, msg= 2015-06-17 11:07:21,767 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:21,768 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50975: callId: 544 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:21,768 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4554 ms ago, cancelled=false, msg= 2015-06-17 11:07:21,971 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:21,971 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50975: callId: 546 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:21,971 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4757 ms ago, cancelled=false, msg= 2015-06-17 11:07:22,174 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:22,174 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50975: callId: 548 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:22,175 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4961 ms ago, cancelled=false, msg= 2015-06-17 11:07:22,376 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:22,376 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50975: callId: 550 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:22,377 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5163 ms ago, cancelled=false, msg= 2015-06-17 11:07:22,580 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:22,580 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50975: callId: 552 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:22,581 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5367 ms ago, cancelled=false, msg= 2015-06-17 11:07:22,581 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:07:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564437214, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:07:22,786 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:22,786 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50975: callId: 554 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:22,789 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:22,789 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50975: callId: 556 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:22,794 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:22,794 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50975: callId: 558 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:22,800 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:22,800 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50975: callId: 560 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:22,811 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:22,812 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50975: callId: 562 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:22,833 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:22,833 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50975: callId: 564 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:22,875 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:22,875 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50975: callId: 566 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:22,977 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:22,977 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50975: callId: 568 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:23,078 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:23,079 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50975: callId: 570 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:23,181 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:23,181 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50975: callId: 572 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:23,284 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:23,284 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50975: callId: 574 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:23,284 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=499 ms ago, cancelled=false, msg= 2015-06-17 11:07:23,485 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:23,485 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50975: callId: 576 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:23,486 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=701 ms ago, cancelled=false, msg= 2015-06-17 11:07:23,687 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:23,687 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50975: callId: 578 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:23,687 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=902 ms ago, cancelled=false, msg= 2015-06-17 11:07:23,889 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:23,889 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50975: callId: 580 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:23,889 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1104 ms ago, cancelled=false, msg= 2015-06-17 11:07:24,091 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:24,091 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50975: callId: 582 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:24,092 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1307 ms ago, cancelled=false, msg= 2015-06-17 11:07:24,294 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:24,294 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50975: callId: 584 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:24,295 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1510 ms ago, cancelled=false, msg= 2015-06-17 11:07:24,497 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:24,497 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50975: callId: 586 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:24,497 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1712 ms ago, cancelled=false, msg= 2015-06-17 11:07:24,698 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:24,699 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50975: callId: 588 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:24,699 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1914 ms ago, cancelled=false, msg= 2015-06-17 11:07:24,901 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:24,901 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50975: callId: 590 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:24,902 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2117 ms ago, cancelled=false, msg= 2015-06-17 11:07:25,104 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:25,104 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50975: callId: 592 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:25,104 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2319 ms ago, cancelled=false, msg= 2015-06-17 11:07:25,305 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:25,306 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50975: callId: 594 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:25,306 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2521 ms ago, cancelled=false, msg= 2015-06-17 11:07:25,509 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:25,509 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50975: callId: 596 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:25,510 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2725 ms ago, cancelled=false, msg= 2015-06-17 11:07:25,701 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(197): executing master kill 2015-06-17 11:07:25,701 INFO [MasterRestartThread] hbase.MiniHBaseCluster(422): Aborting 10.0.0.110,50975,1434564429984 2015-06-17 11:07:25,701 FATAL [MasterRestartThread] master.HMaster(1953): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2015-06-17 11:07:25,702 FATAL [MasterRestartThread] master.HMaster(1956): Aborting for tests java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:423) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:270) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$5.run(TestMasterRestartWithProcedures.java:198) 2015-06-17 11:07:25,702 INFO [MasterRestartThread] regionserver.HRegionServer(1859): STOPPED: Aborting for tests 2015-06-17 11:07:25,702 INFO [MasterRestartThread] hbase.LocalHBaseCluster(379): Waiting on 10.0.0.110,50975,1434564429984 2015-06-17 11:07:25,702 INFO [M:0;10.0.0.110:50975] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2015-06-17 11:07:25,702 INFO [M:0;10.0.0.110:50975] regionserver.HeapMemoryManager(201): Stoping HeapMemoryTuner chore. 2015-06-17 11:07:25,702 INFO [SplitLogWorker-10.0.0.110:50975] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2015-06-17 11:07:25,702 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2015-06-17 11:07:25,702 INFO [SplitLogWorker-10.0.0.110:50975] regionserver.SplitLogWorker(155): SplitLogWorker 10.0.0.110,50975,1434564429984 exiting 2015-06-17 11:07:25,702 INFO [M:0;10.0.0.110:50975] procedure2.ProcedureExecutor(490): Stopping the procedure executor 2015-06-17 11:07:25,702 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2015-06-17 11:07:25,704 INFO [M:0;10.0.0.110:50975] wal.WALProcedureStore(159): Stopping the WAL Procedure Store 2015-06-17 11:07:25,710 INFO [IPC Server handler 4 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741861_1037{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 0 2015-06-17 11:07:25,711 INFO [M:0;10.0.0.110:50975] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2015-06-17 11:07:25,711 INFO [M:0;10.0.0.110:50975] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2015-06-17 11:07:25,711 INFO [M:0;10.0.0.110:50975] regionserver.HRegionServer(1021): stopping server 10.0.0.110,50975,1434564429984 2015-06-17 11:07:25,711 DEBUG [M:0;10.0.0.110:50975] zookeeper.MetaTableLocator(617): Stopping MetaTableLocator 2015-06-17 11:07:25,711 DEBUG [RS_CLOSE_REGION-10.0.0.110:50975-0] handler.CloseRegionHandler(90): Processing close of hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:07:25,712 DEBUG [RS_CLOSE_REGION-10.0.0.110:50975-0] regionserver.HRegion(1365): Closing hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.: disabling compactions & flushes 2015-06-17 11:07:25,712 DEBUG [RS_CLOSE_REGION-10.0.0.110:50975-0] regionserver.HRegion(1392): Updates disabled for region hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:07:25,712 INFO [M:0;10.0.0.110:50975] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a232000f 2015-06-17 11:07:25,713 INFO [StoreCloserThread-hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.-1] regionserver.HStore(870): Closed info 2015-06-17 11:07:25,713 DEBUG [M:0;10.0.0.110:50975] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:07:25,713 INFO [M:0;10.0.0.110:50975] regionserver.CompactSplitThread(394): Waiting for Split Thread to finish... 2015-06-17 11:07:25,714 INFO [M:0;10.0.0.110:50975] regionserver.CompactSplitThread(394): Waiting for Merge Thread to finish... 2015-06-17 11:07:25,714 DEBUG [RpcServer.reader=2,bindAddress=10.0.0.110,port=50975] ipc.RpcServer$Listener(872): RpcServer.listener,port=50975: DISCONNECTING client 10.0.0.110:50987 because read count=-1. Number of active connections: 3 2015-06-17 11:07:25,714 INFO [M:0;10.0.0.110:50975] regionserver.CompactSplitThread(394): Waiting for Large Compaction Thread to finish... 2015-06-17 11:07:25,714 DEBUG [AsyncRpcChannel-pool2-t10] ipc.AsyncRpcChannel$8(574): IPC Client (-352429713) connection to /10.0.0.110:50975 from ndimiduk.hfs.2: closed 2015-06-17 11:07:25,714 DEBUG [AsyncRpcChannel-pool2-t11] ipc.AsyncRpcChannel$8(574): IPC Client (2136356619) connection to /10.0.0.110:50903 from ndimiduk.hfs.2: closed 2015-06-17 11:07:25,714 INFO [M:0;10.0.0.110:50975] regionserver.CompactSplitThread(394): Waiting for Small Compaction Thread to finish... 2015-06-17 11:07:25,715 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:25,715 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=50975: callId: 598 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:25,715 DEBUG [RS_CLOSE_META-10.0.0.110:50975-0] handler.CloseRegionHandler(90): Processing close of hbase:meta,,1.1588230740 2015-06-17 11:07:25,715 INFO [M:0;10.0.0.110:50975] regionserver.HRegionServer(1243): Waiting on 2 regions to close 2015-06-17 11:07:25,715 DEBUG [RS_CLOSE_META-10.0.0.110:50975-0] regionserver.HRegion(1365): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2015-06-17 11:07:25,715 DEBUG [M:0;10.0.0.110:50975] regionserver.HRegionServer(1247): {9ccc7b1e8942a5472c9cf2d844c2fff0=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., 1588230740=hbase:meta,,1.1588230740} 2015-06-17 11:07:25,715 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2930 ms ago, cancelled=false, msg= 2015-06-17 11:07:25,715 DEBUG [RS_CLOSE_META-10.0.0.110:50975-0] regionserver.HRegion(1392): Updates disabled for region hbase:meta,,1.1588230740 2015-06-17 11:07:25,716 INFO [RS_CLOSE_META-10.0.0.110:50975-0] regionserver.HRegion(2111): Flushing 2/2 column families, memstore=1.95 KB 2015-06-17 11:07:25,719 DEBUG [RS_CLOSE_REGION-10.0.0.110:50975-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/recovered.edits/14.seqid to file, newSeqId=14, maxSeqId=12 2015-06-17 11:07:25,719 INFO [RS_CLOSE_REGION-10.0.0.110:50975-0] regionserver.HRegion(1486): Closed hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:07:25,720 DEBUG [RS_CLOSE_REGION-10.0.0.110:50975-0] handler.CloseRegionHandler(122): Closed hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:07:25,724 INFO [IPC Server handler 6 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741862_1038{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 0 2015-06-17 11:07:25,724 INFO [RS_CLOSE_META-10.0.0.110:50975-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=40000117, memsize=1.4 K, hasBloomFilter=false, into tmp file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/7acd1d3a04d44ab0a2c087c8d9edde78 2015-06-17 11:07:25,737 INFO [IPC Server handler 2 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741863_1039{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 4851 2015-06-17 11:07:25,919 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:25,919 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=50975: callId: 600 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:25,920 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3135 ms ago, cancelled=false, msg= 2015-06-17 11:07:26,027 INFO [10.0.0.110,50975,1434564429984_splitLogManager__ChoreService_1] hbase.ScheduledChore(179): Chore: SplitLogManager Timeout Monitor was stopped 2015-06-17 11:07:26,117 INFO [10.0.0.110,50975,1434564429984_ChoreService_1] hbase.ScheduledChore(179): Chore: 10.0.0.110,50975,1434564429984-MemstoreFlusherChore was stopped 2015-06-17 11:07:26,121 INFO [master//10.0.0.110:0.logRoller] regionserver.LogRoller(159): LogRoller exiting. 2015-06-17 11:07:26,121 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(146): master//10.0.0.110:0.leaseChecker closing leases 2015-06-17 11:07:26,121 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(149): master//10.0.0.110:0.leaseChecker closed leases 2015-06-17 11:07:26,124 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:26,124 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=50975: callId: 602 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:26,125 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3340 ms ago, cancelled=false, msg= 2015-06-17 11:07:26,139 INFO [RS_CLOSE_META-10.0.0.110:50975-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=40000117, memsize=552, hasBloomFilter=false, into tmp file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/329b922a4c424a6b9d97b18b74f7f17e 2015-06-17 11:07:26,149 DEBUG [RS_CLOSE_META-10.0.0.110:50975-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/7acd1d3a04d44ab0a2c087c8d9edde78 as hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/7acd1d3a04d44ab0a2c087c8d9edde78 2015-06-17 11:07:26,156 INFO [RS_CLOSE_META-10.0.0.110:50975-0] regionserver.HStore(963): Added hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/7acd1d3a04d44ab0a2c087c8d9edde78, entries=6, sequenceid=40000117, filesize=5.3 K 2015-06-17 11:07:26,157 DEBUG [RS_CLOSE_META-10.0.0.110:50975-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/329b922a4c424a6b9d97b18b74f7f17e as hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/329b922a4c424a6b9d97b18b74f7f17e 2015-06-17 11:07:26,164 INFO [RS_CLOSE_META-10.0.0.110:50975-0] regionserver.HStore(963): Added hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/329b922a4c424a6b9d97b18b74f7f17e, entries=3, sequenceid=40000117, filesize=4.7 K 2015-06-17 11:07:26,165 INFO [RS_CLOSE_META-10.0.0.110:50975-0] regionserver.HRegion(2400): Finished memstore flush of ~1.95 KB/1992, currentsize=0 B/0 for region hbase:meta,,1.1588230740 in 449ms, sequenceid=40000117, compaction requested=false 2015-06-17 11:07:26,166 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(870): Closed info 2015-06-17 11:07:26,167 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(870): Closed table 2015-06-17 11:07:26,168 INFO [RS_OPEN_META-10.0.0.110:50975-0-MetaLogRoller] regionserver.LogRoller(159): LogRoller exiting. 2015-06-17 11:07:26,172 DEBUG [RS_CLOSE_META-10.0.0.110:50975-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/recovered.edits/40000120.seqid to file, newSeqId=40000120, maxSeqId=40000104 2015-06-17 11:07:26,173 DEBUG [RS_CLOSE_META-10.0.0.110:50975-0] coprocessor.CoprocessorHost(269): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2015-06-17 11:07:26,173 INFO [RS_CLOSE_META-10.0.0.110:50975-0] regionserver.HRegion(1486): Closed hbase:meta,,1.1588230740 2015-06-17 11:07:26,173 DEBUG [RS_CLOSE_META-10.0.0.110:50975-0] handler.CloseRegionHandler(122): Closed hbase:meta,,1.1588230740 2015-06-17 11:07:26,317 INFO [M:0;10.0.0.110:50975] regionserver.HRegionServer(1049): stopping server 10.0.0.110,50975,1434564429984; all regions closed. 2015-06-17 11:07:26,317 DEBUG [M:0;10.0.0.110:50975] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50975,1434564429984 2015-06-17 11:07:26,320 INFO [IPC Server handler 2 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741860_1036{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 83 2015-06-17 11:07:26,324 DEBUG [M:0;10.0.0.110:50975] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:07:26,324 INFO [M:0;10.0.0.110:50975] wal.FSHLog(977): Closed WAL: FSHLog 10.0.0.110%2C50975%2C1434564429984..meta:.meta(num 1434564430162) 2015-06-17 11:07:26,324 DEBUG [M:0;10.0.0.110:50975] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50975,1434564429984 2015-06-17 11:07:26,326 INFO [IPC Server handler 1 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741859_1035{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 679 2015-06-17 11:07:26,328 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:26,328 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=50975: callId: 604 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:26,328 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3543 ms ago, cancelled=false, msg= 2015-06-17 11:07:26,400 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=350, started=15952 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:26,534 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:26,534 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=50975] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=50975: callId: 606 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:50980 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:26,535 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3750 ms ago, cancelled=false, msg= 2015-06-17 11:07:26,600 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=350, started=16152 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:26,732 DEBUG [M:0;10.0.0.110:50975] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:07:26,732 INFO [M:0;10.0.0.110:50975] wal.FSHLog(977): Closed WAL: FSHLog 10.0.0.110%2C50975%2C1434564429984.default:(num 1434564430107) 2015-06-17 11:07:26,732 DEBUG [M:0;10.0.0.110:50975] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:07:26,732 INFO [M:0;10.0.0.110:50975] regionserver.Leases(146): M:0;10.0.0.110:50975 closing leases 2015-06-17 11:07:26,732 INFO [M:0;10.0.0.110:50975] regionserver.Leases(149): M:0;10.0.0.110:50975 closed leases 2015-06-17 11:07:26,732 INFO [M:0;10.0.0.110:50975] hbase.ChoreService(302): Chore service for: 10.0.0.110,50975,1434564429984 had [[ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-10.0.0.110:50975 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.0.0.110,50975,1434564429984-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.0.0.110,50975,1434564429984-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region 10.0.0.110,50975,1434564429984 Period: 120000 Unit: MILLISECONDS]] on shutdown 2015-06-17 11:07:26,735 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/10.0.0.110,50975,1434564429984 2015-06-17 11:07:26,735 INFO [M:0;10.0.0.110:50975] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320012 2015-06-17 11:07:26,736 DEBUG [M:0;10.0.0.110:50975] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:07:26,736 DEBUG [M:0;10.0.0.110:50975] master.HMaster(1074): Stopping service threads 2015-06-17 11:07:26,737 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:07:26,737 INFO [M:0;10.0.0.110:50975] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320010 2015-06-17 11:07:26,737 DEBUG [Thread-199] zookeeper.ZKUtil(745): hconnection-0x51e5f22a-0x14e02b3a2320008, quorum=localhost:51848, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2015-06-17 11:07:26,737 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:07:26,738 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3953 ms ago, cancelled=false, msg= 2015-06-17 11:07:26,738 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(488): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:07:26,738 DEBUG [M:0;10.0.0.110:50975] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:07:26,738 DEBUG [main-EventThread] zookeeper.ZKUtil(488): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:07:26,738 INFO [M:0;10.0.0.110:50975] hbase.ChoreService(302): Chore service for: 10.0.0.110,50975,1434564429984_splitLogManager_ had [] on shutdown 2015-06-17 11:07:26,738 INFO [M:0;10.0.0.110:50975] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2015-06-17 11:07:26,738 INFO [M:0;10.0.0.110:50975] ipc.RpcServer(2219): Stopping server on 50975 2015-06-17 11:07:26,739 INFO [RpcServer.listener,port=50975] ipc.RpcServer$Listener(776): RpcServer.listener,port=50975: stopping 2015-06-17 11:07:26,739 INFO [RpcServer.responder] ipc.RpcServer$Responder(1018): RpcServer.responder: stopped 2015-06-17 11:07:26,739 INFO [RpcServer.responder] ipc.RpcServer$Responder(921): RpcServer.responder: stopping 2015-06-17 11:07:26,740 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,50975,1434564429984 2015-06-17 11:07:26,740 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,50975,1434564429984 2015-06-17 11:07:26,740 INFO [MasterRestartThread-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [10.0.0.110,50975,1434564429984] 2015-06-17 11:07:26,741 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:50975-0x14e02b3a232000e, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:07:26,741 INFO [M:0;10.0.0.110:50975] regionserver.HRegionServer(1093): stopping server 10.0.0.110,50975,1434564429984; zookeeper connection closed. 2015-06-17 11:07:26,741 INFO [M:0;10.0.0.110:50975] regionserver.HRegionServer(1096): M:0;10.0.0.110:50975 exiting 2015-06-17 11:07:26,741 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(200): executing master restart 2015-06-17 11:07:26,741 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:26,741 INFO [main-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(121): /hbase/rs/10.0.0.110,50975,1434564429984 znode expired, triggering replicatorRemoved event 2015-06-17 11:07:26,741 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:07:26,742 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:26,762 INFO [MasterRestartThread] client.ConnectionUtils(105): master//10.0.0.110:0 server-side HConnection retries=350 2015-06-17 11:07:26,762 INFO [MasterRestartThread] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2015-06-17 11:07:26,765 INFO [MasterRestartThread] ipc.RpcServer$Listener(598): master//10.0.0.110:0: started 10 reader(s). 2015-06-17 11:07:26,767 INFO [MasterRestartThread] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:07:26,768 INFO [MasterRestartThread] zookeeper.RecoverableZooKeeper(121): Process identifier=master:51002 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:07:26,771 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:510020x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:07:26,772 DEBUG [MasterRestartThread] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-17 11:07:26,773 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(578): master:51002-0x14e02b3a2320013 connected 2015-06-17 11:07:26,775 DEBUG [MasterRestartThread] zookeeper.ZKUtil(488): master:51002-0x14e02b3a2320013, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:07:26,776 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:51002-0x14e02b3a2320013, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2015-06-17 11:07:26,776 INFO [RpcServer.responder] ipc.RpcServer$Responder(917): RpcServer.responder: starting 2015-06-17 11:07:26,776 INFO [RpcServer.listener,port=51002] ipc.RpcServer$Listener(729): RpcServer.listener,port=51002: starting 2015-06-17 11:07:26,776 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=0 queue=0 2015-06-17 11:07:26,776 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=1 queue=0 2015-06-17 11:07:26,777 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=2 queue=0 2015-06-17 11:07:26,777 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=3 queue=0 2015-06-17 11:07:26,777 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=4 queue=0 2015-06-17 11:07:26,777 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=0 queue=0 2015-06-17 11:07:26,777 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=1 queue=1 2015-06-17 11:07:26,777 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=2 queue=0 2015-06-17 11:07:26,777 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=3 queue=1 2015-06-17 11:07:26,777 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=4 queue=0 2015-06-17 11:07:26,778 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=5 queue=1 2015-06-17 11:07:26,778 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=6 queue=0 2015-06-17 11:07:26,778 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=7 queue=1 2015-06-17 11:07:26,778 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=8 queue=0 2015-06-17 11:07:26,778 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=9 queue=1 2015-06-17 11:07:26,778 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=10 queue=0 2015-06-17 11:07:26,778 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=11 queue=1 2015-06-17 11:07:26,778 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=12 queue=0 2015-06-17 11:07:26,778 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=13 queue=1 2015-06-17 11:07:26,779 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=14 queue=0 2015-06-17 11:07:26,779 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=15 queue=1 2015-06-17 11:07:26,779 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=16 queue=0 2015-06-17 11:07:26,779 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=17 queue=1 2015-06-17 11:07:26,779 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=18 queue=0 2015-06-17 11:07:26,779 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=19 queue=1 2015-06-17 11:07:26,779 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=0 queue=0 2015-06-17 11:07:26,779 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=1 queue=0 2015-06-17 11:07:26,780 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=2 queue=0 2015-06-17 11:07:26,780 INFO [MasterRestartThread] master.HMaster(349): hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034, hbase.cluster.distributed=false 2015-06-17 11:07:26,780 INFO [MasterRestartThread] master.HMaster(1512): Adding backup master ZNode /hbase/backup-masters/10.0.0.110,51002,1434564446765 2015-06-17 11:07:26,781 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:51002-0x14e02b3a2320013, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/10.0.0.110,51002,1434564446765 2015-06-17 11:07:26,782 INFO [M:0;10.0.0.110:51002] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x7cfeace3 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:07:26,782 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51002-0x14e02b3a2320013, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:07:26,782 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:07:26,782 DEBUG [10.0.0.110:51002.activeMasterManager] zookeeper.ZKUtil(486): master:51002-0x14e02b3a2320013, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:07:26,783 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(486): master:51002-0x14e02b3a2320013, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:07:26,783 INFO [10.0.0.110:51002.activeMasterManager] master.ActiveMasterManager(170): Deleting ZNode for /hbase/backup-masters/10.0.0.110,51002,1434564446765 from backup master directory 2015-06-17 11:07:26,783 DEBUG [MasterRestartThread-EventThread] master.ActiveMasterManager(126): A master is now available 2015-06-17 11:07:26,783 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51002-0x14e02b3a2320013, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/10.0.0.110,51002,1434564446765 2015-06-17 11:07:26,783 WARN [10.0.0.110:51002.activeMasterManager] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:07:26,784 INFO [10.0.0.110:51002.activeMasterManager] master.ActiveMasterManager(179): Registered Active Master=10.0.0.110,51002,1434564446765 2015-06-17 11:07:26,784 DEBUG [M:0;10.0.0.110:51002-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x7cfeace30x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:07:26,784 DEBUG [M:0;10.0.0.110:51002] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@c518734, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:07:26,784 DEBUG [M:0;10.0.0.110:51002] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:07:26,785 DEBUG [M:0;10.0.0.110:51002] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:07:26,785 DEBUG [M:0;10.0.0.110:51002-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x7cfeace3-0x14e02b3a2320014 connected 2015-06-17 11:07:26,786 INFO [M:0;10.0.0.110:51002] regionserver.HRegionServer(774): ClusterId : b7c4b21b-aab2-44b4-8562-2b61af22ad6c 2015-06-17 11:07:26,786 DEBUG [M:0;10.0.0.110:51002] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2015-06-17 11:07:26,786 DEBUG [M:0;10.0.0.110:51002] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:07:26,787 DEBUG [M:0;10.0.0.110:51002] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2015-06-17 11:07:26,787 DEBUG [M:0;10.0.0.110:51002] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2015-06-17 11:07:26,787 DEBUG [M:0;10.0.0.110:51002] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:07:26,788 DEBUG [M:0;10.0.0.110:51002] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2015-06-17 11:07:26,788 INFO [M:0;10.0.0.110:51002] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2015-06-17 11:07:26,789 INFO [M:0;10.0.0.110:51002] compactions.PressureAwareCompactionThroughputController(175): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2015-06-17 11:07:26,789 INFO [M:0;10.0.0.110:51002] regionserver.HRegionServer$CompactionChecker(1504): CompactionChecker runs every 1sec 2015-06-17 11:07:26,789 DEBUG [M:0;10.0.0.110:51002] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bea96de, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=10.0.0.110/10.0.0.110:0 2015-06-17 11:07:26,789 DEBUG [M:0;10.0.0.110:51002] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:07:26,789 DEBUG [M:0;10.0.0.110:51002] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:07:26,790 DEBUG [M:0;10.0.0.110:51002] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:M:0;10.0.0.110:51002 2015-06-17 11:07:26,790 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:07:26,791 DEBUG [M:0;10.0.0.110:51002] zookeeper.ZKUtil(486): master:51002-0x14e02b3a2320013, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51002,1434564446765 2015-06-17 11:07:26,791 INFO [M:0;10.0.0.110:51002] regionserver.RegionServerCoprocessorHost(65): System coprocessor loading is enabled 2015-06-17 11:07:26,791 INFO [M:0;10.0.0.110:51002] regionserver.RegionServerCoprocessorHost(66): Table coprocessor loading is enabled 2015-06-17 11:07:26,791 INFO [M:0;10.0.0.110:51002] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,51002,1434564446765 with port=51002, startcode=1434564446765 2015-06-17 11:07:26,791 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:26,791 DEBUG [M:0;10.0.0.110:51002] regionserver.HRegionServer(2295): Master is not running yet 2015-06-17 11:07:26,791 WARN [M:0;10.0.0.110:51002] regionserver.HRegionServer(900): reportForDuty failed; sleeping and then retrying. 2015-06-17 11:07:26,791 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51002,1434564446765 2015-06-17 11:07:26,796 DEBUG [10.0.0.110:51002.activeMasterManager] util.FSTableDescriptors(807): Current tableInfoPath = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2015-06-17 11:07:26,800 DEBUG [10.0.0.110:51002.activeMasterManager] util.FSTableDescriptors(811): TableInfo already exists.. Skipping creation 2015-06-17 11:07:26,802 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=350, started=16354 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:26,804 INFO [10.0.0.110:51002.activeMasterManager] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:07:26,804 DEBUG [10.0.0.110:51002.activeMasterManager] coordination.ZKSplitLogManagerCoordination(885): Distributed log replay=true 2015-06-17 11:07:26,805 INFO [10.0.0.110:51002.activeMasterManager] coordination.ZKSplitLogManagerCoordination(599): Found 0 orphan tasks and 0 rescan nodes 2015-06-17 11:07:26,805 DEBUG [10.0.0.110:51002.activeMasterManager] util.FSTableDescriptors(222): Fetching table descriptors from the filesystem. 2015-06-17 11:07:26,816 INFO [10.0.0.110:51002.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x185eec9b connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:07:26,818 DEBUG [10.0.0.110:51002.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x185eec9b0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:07:26,819 DEBUG [10.0.0.110:51002.activeMasterManager] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7531108, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:07:26,819 DEBUG [10.0.0.110:51002.activeMasterManager] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:07:26,819 DEBUG [10.0.0.110:51002.activeMasterManager] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:07:26,820 DEBUG [10.0.0.110:51002.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x185eec9b-0x14e02b3a2320015 connected 2015-06-17 11:07:26,821 INFO [10.0.0.110:51002.activeMasterManager] balancer.StochasticLoadBalancer(135): loading config 2015-06-17 11:07:26,821 DEBUG [10.0.0.110:51002.activeMasterManager] zookeeper.ZKUtil(488): master:51002-0x14e02b3a2320013, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2015-06-17 11:07:26,822 DEBUG [10.0.0.110:51002.activeMasterManager] zookeeper.ZKUtil(486): master:51002-0x14e02b3a2320013, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:26,822 DEBUG [10.0.0.110:51002.activeMasterManager] zookeeper.ZKUtil(486): master:51002-0x14e02b3a2320013, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51002,1434564446765 2015-06-17 11:07:26,822 DEBUG [10.0.0.110:51002.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:26,822 DEBUG [10.0.0.110:51002.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,51002,1434564446765 2015-06-17 11:07:26,823 INFO [10.0.0.110:51002.activeMasterManager] master.HMaster(568): Server active/primary master=10.0.0.110,51002,1434564446765, sessionid=0x14e02b3a2320013, setting cluster-up flag (Was=true) 2015-06-17 11:07:26,824 DEBUG [10.0.0.110:51002.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:07:26,824 INFO [10.0.0.110:51002.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2015-06-17 11:07:26,825 DEBUG [10.0.0.110:51002.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,51002,1434564446765 2015-06-17 11:07:26,826 DEBUG [10.0.0.110:51002.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:07:26,826 INFO [10.0.0.110:51002.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2015-06-17 11:07:26,827 DEBUG [10.0.0.110:51002.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,51002,1434564446765 2015-06-17 11:07:26,827 INFO [10.0.0.110:51002.activeMasterManager] master.MasterCoprocessorHost(86): System coprocessor loading is enabled 2015-06-17 11:07:26,827 DEBUG [10.0.0.110:51002.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_OPEN_REGION-10.0.0.110:51002, corePoolSize=5, maxPoolSize=5 2015-06-17 11:07:26,827 DEBUG [10.0.0.110:51002.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_CLOSE_REGION-10.0.0.110:51002, corePoolSize=5, maxPoolSize=5 2015-06-17 11:07:26,827 DEBUG [10.0.0.110:51002.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_SERVER_OPERATIONS-10.0.0.110:51002, corePoolSize=5, maxPoolSize=5 2015-06-17 11:07:26,828 DEBUG [10.0.0.110:51002.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_META_SERVER_OPERATIONS-10.0.0.110:51002, corePoolSize=5, maxPoolSize=5 2015-06-17 11:07:26,828 DEBUG [10.0.0.110:51002.activeMasterManager] executor.ExecutorService(91): Starting executor service name=M_LOG_REPLAY_OPS-10.0.0.110:51002, corePoolSize=10, maxPoolSize=10 2015-06-17 11:07:26,828 DEBUG [10.0.0.110:51002.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_TABLE_OPERATIONS-10.0.0.110:51002, corePoolSize=1, maxPoolSize=1 2015-06-17 11:07:26,828 INFO [10.0.0.110:51002.activeMasterManager] procedure2.ProcedureExecutor(446): Starting procedure executor threads=9 2015-06-17 11:07:26,828 INFO [10.0.0.110:51002.activeMasterManager] wal.WALProcedureStore(214): Starting WAL Procedure Store lease recovery 2015-06-17 11:07:26,829 INFO [10.0.0.110:51002.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log 2015-06-17 11:07:26,829 INFO [10.0.0.110:51002.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log after 0ms 2015-06-17 11:07:26,829 DEBUG [10.0.0.110:51002.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log; isDirectory=false; length=14915; replication=1; blocksize=134217728; modification_time=1434564413352; access_time=1434564405122; owner=ndimiduk; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:07:26,832 INFO [10.0.0.110:51002.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000003.log 2015-06-17 11:07:26,832 INFO [10.0.0.110:51002.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000003.log after 0ms 2015-06-17 11:07:26,832 DEBUG [10.0.0.110:51002.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000003.log; isDirectory=false; length=60; replication=1; blocksize=134217728; modification_time=1434564445711; access_time=1434564430061; owner=ndimiduk.hfs.2; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:07:26,836 DEBUG [10.0.0.110:51002.activeMasterManager] wal.WALProcedureStore(610): Roll new state log: 4 2015-06-17 11:07:26,837 INFO [10.0.0.110:51002.activeMasterManager] wal.WALProcedureStore(237): Lease acquired for flushLogId: 4 2015-06-17 11:07:26,838 INFO [10.0.0.110:51002.activeMasterManager] wal.ProcedureWALFormatReader(150): No active entry found in state log hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000003.log. removing it 2015-06-17 11:07:26,839 DEBUG [10.0.0.110:51002.activeMasterManager] procedure2.ProcedureExecutor$1(278): load procedures maxProcId=7 2015-06-17 11:07:26,840 DEBUG [10.0.0.110:51002.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=RUNNABLE isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=7 state=RUNNABLE 2015-06-17 11:07:26,841 DEBUG [10.0.0.110:51002.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DeleteTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=6 state=FINISHED 2015-06-17 11:07:26,841 DEBUG [10.0.0.110:51002.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:26,841 DEBUG [10.0.0.110:51002.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=5 state=FINISHED 2015-06-17 11:07:26,841 DEBUG [10.0.0.110:51002.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:26,841 DEBUG [10.0.0.110:51002.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: EnableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=4 state=FINISHED 2015-06-17 11:07:26,841 DEBUG [10.0.0.110:51002.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:26,841 DEBUG [10.0.0.110:51002.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=3 state=FINISHED 2015-06-17 11:07:26,841 DEBUG [10.0.0.110:51002.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:26,842 DEBUG [10.0.0.110:51002.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=2 state=FINISHED 2015-06-17 11:07:26,842 DEBUG [10.0.0.110:51002.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:26,842 DEBUG [10.0.0.110:51002.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=hbase:namespace) user=ndimiduk (auth:SIMPLE) id=1 state=FINISHED 2015-06-17 11:07:26,842 DEBUG [10.0.0.110:51002.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:26,842 DEBUG [10.0.0.110:51002.activeMasterManager] wal.WALProcedureStore(652): Remove log: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000003.log 2015-06-17 11:07:26,843 INFO [IPC Server handler 7 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741861_1037 127.0.0.1:50896 2015-06-17 11:07:26,844 DEBUG [10.0.0.110:51002.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2015-06-17 11:07:26,845 INFO [10.0.0.110:51002.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:07:26,846 DEBUG [10.0.0.110:51002.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:07:26,847 DEBUG [10.0.0.110:51002.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/replication/rs already exists 2015-06-17 11:07:26,847 DEBUG [10.0.0.110:51002.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2015-06-17 11:07:26,847 DEBUG [10.0.0.110:51002.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner 2015-06-17 11:07:26,848 DEBUG [10.0.0.110:51002.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): replicationLogCleaner-0x14e02b3a2320016 connected 2015-06-17 11:07:26,848 DEBUG [10.0.0.110:51002.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2015-06-17 11:07:26,848 DEBUG [10.0.0.110:51002.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2015-06-17 11:07:26,848 DEBUG [10.0.0.110:51002.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2015-06-17 11:07:26,848 INFO [10.0.0.110:51002.activeMasterManager] master.ServerManager(1011): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 1, timeout of 4500 ms, interval of 1500 ms. 2015-06-17 11:07:26,848 INFO [M:0;10.0.0.110:51002] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,51002,1434564446765 with port=51002, startcode=1434564446765 2015-06-17 11:07:26,848 INFO [M:0;10.0.0.110:51002] master.ServerManager(459): Registering server=10.0.0.110,51002,1434564446765 2015-06-17 11:07:26,848 INFO [M:0;10.0.0.110:51002] regionserver.HRegionServer(1341): Config from master: hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034 2015-06-17 11:07:26,848 INFO [M:0;10.0.0.110:51002] regionserver.HRegionServer(1341): Config from master: fs.defaultFS=hdfs://localhost:50895 2015-06-17 11:07:26,849 INFO [M:0;10.0.0.110:51002] regionserver.HRegionServer(1341): Config from master: hbase.master.info.port=-1 2015-06-17 11:07:26,849 WARN [M:0;10.0.0.110:51002] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:07:26,849 INFO [M:0;10.0.0.110:51002] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:07:26,849 DEBUG [M:0;10.0.0.110:51002] regionserver.HRegionServer(1603): logdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51002,1434564446765 2015-06-17 11:07:26,852 DEBUG [M:0;10.0.0.110:51002] regionserver.Replication(142): ReplicationStatisticsThread 300 2015-06-17 11:07:26,852 INFO [M:0;10.0.0.110:51002] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:07:26,855 INFO [M:0;10.0.0.110:51002] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.0.0.110%2C51002%2C1434564446765.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51002,1434564446765, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:07:26,862 INFO [M:0;10.0.0.110:51002] wal.FSHLog(1365): Slow sync cost: 3 ms, current pipeline: [] 2015-06-17 11:07:26,862 INFO [M:0;10.0.0.110:51002] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51002,1434564446765/10.0.0.110%2C51002%2C1434564446765.default.1434564446855 2015-06-17 11:07:26,863 INFO [M:0;10.0.0.110:51002] regionserver.MetricsRegionServerWrapperImpl(113): Computing regionserver metrics every 5000 milliseconds 2015-06-17 11:07:26,863 DEBUG [M:0;10.0.0.110:51002] executor.ExecutorService(91): Starting executor service name=RS_OPEN_REGION-10.0.0.110:51002, corePoolSize=3, maxPoolSize=3 2015-06-17 11:07:26,863 DEBUG [M:0;10.0.0.110:51002] executor.ExecutorService(91): Starting executor service name=RS_OPEN_META-10.0.0.110:51002, corePoolSize=1, maxPoolSize=1 2015-06-17 11:07:26,863 DEBUG [M:0;10.0.0.110:51002] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_REGION-10.0.0.110:51002, corePoolSize=3, maxPoolSize=3 2015-06-17 11:07:26,864 DEBUG [M:0;10.0.0.110:51002] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_META-10.0.0.110:51002, corePoolSize=1, maxPoolSize=1 2015-06-17 11:07:26,864 DEBUG [M:0;10.0.0.110:51002] executor.ExecutorService(91): Starting executor service name=RS_LOG_REPLAY_OPS-10.0.0.110:51002, corePoolSize=2, maxPoolSize=2 2015-06-17 11:07:26,864 DEBUG [M:0;10.0.0.110:51002] executor.ExecutorService(91): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-10.0.0.110:51002, corePoolSize=3, maxPoolSize=3 2015-06-17 11:07:26,865 DEBUG [M:0;10.0.0.110:51002] zookeeper.ZKUtil(486): master:51002-0x14e02b3a2320013, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:26,865 DEBUG [M:0;10.0.0.110:51002] zookeeper.ZKUtil(486): master:51002-0x14e02b3a2320013, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51002,1434564446765 2015-06-17 11:07:26,865 INFO [M:0;10.0.0.110:51002] regionserver.ReplicationSourceManager(229): Current list of replicators: [10.0.0.110,50905,1434564404764, 10.0.0.110,51002,1434564446765] other RSs: [10.0.0.110,50905,1434564404764, 10.0.0.110,51002,1434564446765] 2015-06-17 11:07:26,878 INFO [M:0;10.0.0.110:51002] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x6dac3e37 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:07:26,880 DEBUG [M:0;10.0.0.110:51002-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x6dac3e370x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:07:26,881 DEBUG [M:0;10.0.0.110:51002] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodecWithTags@6cfee4d2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:07:26,881 DEBUG [M:0;10.0.0.110:51002] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:07:26,881 DEBUG [M:0;10.0.0.110:51002] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:07:26,882 DEBUG [M:0;10.0.0.110:51002-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x6dac3e37-0x14e02b3a2320017 connected 2015-06-17 11:07:26,895 INFO [M:0;10.0.0.110:51002] regionserver.HeapMemoryManager(190): Starting HeapMemoryTuner chore. 2015-06-17 11:07:26,895 INFO [SplitLogWorker-10.0.0.110:51002] regionserver.SplitLogWorker(134): SplitLogWorker 10.0.0.110,51002,1434564446765 starting 2015-06-17 11:07:26,895 INFO [M:0;10.0.0.110:51002] regionserver.HRegionServer(1363): Serving as 10.0.0.110,51002,1434564446765, RpcServer on 10.0.0.110/10.0.0.110:51002, sessionid=0x14e02b3a2320013 2015-06-17 11:07:26,896 DEBUG [M:0;10.0.0.110:51002] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2015-06-17 11:07:26,896 DEBUG [M:0;10.0.0.110:51002] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager 10.0.0.110,51002,1434564446765 2015-06-17 11:07:26,896 DEBUG [M:0;10.0.0.110:51002] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,51002,1434564446765' 2015-06-17 11:07:26,896 DEBUG [M:0;10.0.0.110:51002] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2015-06-17 11:07:26,896 DEBUG [M:0;10.0.0.110:51002] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2015-06-17 11:07:26,896 DEBUG [M:0;10.0.0.110:51002] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2015-06-17 11:07:26,896 DEBUG [M:0;10.0.0.110:51002] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2015-06-17 11:07:26,896 DEBUG [M:0;10.0.0.110:51002] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager 10.0.0.110,51002,1434564446765 2015-06-17 11:07:26,897 DEBUG [M:0;10.0.0.110:51002] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,51002,1434564446765' 2015-06-17 11:07:26,897 DEBUG [M:0;10.0.0.110:51002] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2015-06-17 11:07:26,897 DEBUG [M:0;10.0.0.110:51002] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2015-06-17 11:07:26,897 DEBUG [M:0;10.0.0.110:51002] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2015-06-17 11:07:26,897 INFO [M:0;10.0.0.110:51002] quotas.RegionServerQuotaManager(62): Quota support disabled 2015-06-17 11:07:26,898 INFO [10.0.0.110:51002.activeMasterManager] master.ServerManager(1028): Finished waiting for region servers count to settle; checked in 1, slept for 50 ms, expecting minimum of 1, maximum of 1, master is running 2015-06-17 11:07:26,899 INFO [10.0.0.110:51002.activeMasterManager] master.ServerManager(459): Registering server=10.0.0.110,50905,1434564404764 2015-06-17 11:07:26,899 INFO [10.0.0.110:51002.activeMasterManager] master.HMaster(665): Registered server found up in zk but who has not yet reported in: 10.0.0.110,50905,1434564404764 2015-06-17 11:07:26,901 INFO [10.0.0.110:51002.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50905,1434564404764 belongs to an existing region server 2015-06-17 11:07:26,902 INFO [10.0.0.110:51002.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51002,1434564446765 belongs to an existing region server 2015-06-17 11:07:26,903 WARN [10.0.0.110:51002.activeMasterManager] master.RegionStates(339): Open region's last host null should be the same as the current one 10.0.0.110,50975,1434564429984, ignored the last and used the current one 2015-06-17 11:07:26,939 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:07:26,941 DEBUG [AsyncRpcChannel-pool2-t14] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service MasterService, sasl=false 2015-06-17 11:07:26,941 DEBUG [RpcServer.listener,port=51002] ipc.RpcServer$Listener(840): RpcServer.listener,port=51002: connection from 10.0.0.110:51011; # active connections: 1 2015-06-17 11:07:26,942 INFO [RpcServer.reader=1,bindAddress=10.0.0.110,port=51002] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51011 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:07:26,943 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:26,943 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51002: callId: 610 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:26,943 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4158 ms ago, cancelled=false, msg= 2015-06-17 11:07:27,003 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=350, started=16555 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:27,145 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:27,146 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51002: callId: 612 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:27,146 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4361 ms ago, cancelled=false, msg= 2015-06-17 11:07:27,204 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=350, started=16756 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:27,347 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:27,347 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51002: callId: 614 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:27,348 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4563 ms ago, cancelled=false, msg= 2015-06-17 11:07:27,404 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=350, started=16956 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:27,549 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:27,549 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51002: callId: 616 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:27,550 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4765 ms ago, cancelled=false, msg= 2015-06-17 11:07:27,607 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=350, started=17159 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:27,751 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:27,751 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51002: callId: 618 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:27,751 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4966 ms ago, cancelled=false, msg= 2015-06-17 11:07:27,808 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=350, started=17360 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:27,952 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:27,953 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51002: callId: 620 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:27,953 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5168 ms ago, cancelled=false, msg= 2015-06-17 11:07:28,009 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=350, started=17561 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:28,154 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:28,154 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51002: callId: 622 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:28,155 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5370 ms ago, cancelled=false, msg= 2015-06-17 11:07:28,155 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:07:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: Can't get master address from ZooKeeper; znode data == null Wed Jun 17 11:07:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564442785, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:07:28,209 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=350, started=17761 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:28,358 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:28,359 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51002: callId: 624 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:28,362 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:28,362 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51002: callId: 626 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:28,367 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:28,368 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51002: callId: 628 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:28,374 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:28,374 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51002: callId: 630 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:28,386 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:28,386 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51002: callId: 632 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:28,407 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:28,407 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51002: callId: 634 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:28,410 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=350, started=17962 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:28,417 DEBUG [AsyncRpcChannel-pool2-t15] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2015-06-17 11:07:28,417 DEBUG [RpcServer.listener,port=51002] ipc.RpcServer$Listener(840): RpcServer.listener,port=51002: connection from 10.0.0.110:51012; # active connections: 2 2015-06-17 11:07:28,418 INFO [RpcServer.reader=2,bindAddress=10.0.0.110,port=51002] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51012 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:07:28,449 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:28,450 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51002: callId: 636 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:28,551 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:28,551 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51002: callId: 638 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:28,612 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=350, started=18164 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:28,653 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:28,653 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51002: callId: 640 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:28,755 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:28,755 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51002: callId: 642 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:28,812 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=350, started=18364 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:28,856 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:28,856 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51002: callId: 644 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:28,857 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=500 ms ago, cancelled=false, msg= 2015-06-17 11:07:29,013 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=350, started=18565 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:29,059 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:29,059 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51002: callId: 646 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:29,060 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=703 ms ago, cancelled=false, msg= 2015-06-17 11:07:29,214 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=350, started=18766 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:29,261 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:29,261 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51002: callId: 648 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:29,262 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=904 ms ago, cancelled=false, msg= 2015-06-17 11:07:29,415 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=350, started=18967 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:29,465 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:29,465 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51002: callId: 650 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:29,465 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1108 ms ago, cancelled=false, msg= 2015-06-17 11:07:29,617 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=350, started=19169 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:29,667 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:29,667 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51002: callId: 652 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:29,668 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1311 ms ago, cancelled=false, msg= 2015-06-17 11:07:29,817 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=350, started=19369 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:29,869 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:29,869 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51002: callId: 654 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:29,870 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1513 ms ago, cancelled=false, msg= 2015-06-17 11:07:30,017 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=350, started=19569 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:30,071 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:30,071 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51002: callId: 656 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:30,072 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1715 ms ago, cancelled=false, msg= 2015-06-17 11:07:30,219 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=350, started=19771 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:30,274 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:30,274 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51002: callId: 658 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:30,274 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1917 ms ago, cancelled=false, msg= 2015-06-17 11:07:30,421 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=350, started=19973 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:30,476 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:30,476 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51002: callId: 660 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:30,477 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2120 ms ago, cancelled=false, msg= 2015-06-17 11:07:30,622 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=350, started=20174 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:30,678 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:30,678 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51002: callId: 662 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:30,678 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2321 ms ago, cancelled=false, msg= 2015-06-17 11:07:30,822 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=350, started=20374 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:30,881 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:30,882 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51002: callId: 664 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:30,882 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2525 ms ago, cancelled=false, msg= 2015-06-17 11:07:31,023 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=350, started=20575 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:31,084 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:31,084 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51002: callId: 666 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:31,085 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2728 ms ago, cancelled=false, msg= 2015-06-17 11:07:31,226 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=350, started=20777 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:31,287 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:31,287 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51002: callId: 668 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:31,287 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2930 ms ago, cancelled=false, msg= 2015-06-17 11:07:31,426 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=35, retries=350, started=20978 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:31,488 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:31,488 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51002: callId: 670 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:31,489 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3132 ms ago, cancelled=false, msg= 2015-06-17 11:07:31,627 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=36, retries=350, started=21179 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:31,691 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:31,691 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51002: callId: 672 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:31,692 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3335 ms ago, cancelled=false, msg= 2015-06-17 11:07:31,827 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=37, retries=350, started=21379 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:31,895 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:31,895 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51002: callId: 674 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:31,895 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3538 ms ago, cancelled=false, msg= 2015-06-17 11:07:32,028 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=38, retries=350, started=21580 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:32,098 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:32,098 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51002: callId: 676 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:32,098 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3741 ms ago, cancelled=false, msg= 2015-06-17 11:07:32,230 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=39, retries=350, started=21782 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:32,300 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:32,300 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51002: callId: 678 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:32,301 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3944 ms ago, cancelled=false, msg= 2015-06-17 11:07:32,430 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=40, retries=350, started=21982 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:32,503 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:32,503 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51002: callId: 680 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:32,503 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4146 ms ago, cancelled=false, msg= 2015-06-17 11:07:32,631 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=41, retries=350, started=22183 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:32,704 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:32,705 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51002: callId: 682 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:32,705 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4348 ms ago, cancelled=false, msg= 2015-06-17 11:07:32,832 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=42, retries=350, started=22384 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:32,907 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:32,907 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51002: callId: 684 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:32,908 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4551 ms ago, cancelled=false, msg= 2015-06-17 11:07:33,033 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=43, retries=350, started=22585 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:33,110 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:33,110 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51002: callId: 686 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:33,110 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4753 ms ago, cancelled=false, msg= 2015-06-17 11:07:33,234 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=44, retries=350, started=22786 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:33,313 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:33,313 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51002: callId: 688 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:33,313 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4956 ms ago, cancelled=false, msg= 2015-06-17 11:07:33,435 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=45, retries=350, started=22987 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:33,515 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:33,516 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51002: callId: 690 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:33,516 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5159 ms ago, cancelled=false, msg= 2015-06-17 11:07:33,636 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=46, retries=350, started=23188 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:33,718 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:33,718 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51002: callId: 692 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:33,719 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5362 ms ago, cancelled=false, msg= 2015-06-17 11:07:33,719 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564448357, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:07:33,836 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=47, retries=350, started=23388 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:33,921 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:33,921 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51002: callId: 694 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:33,925 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:33,925 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51002: callId: 696 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:33,929 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:33,929 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51002: callId: 698 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:33,935 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:33,935 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51002: callId: 700 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:33,946 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:33,946 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51002: callId: 702 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:33,967 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:33,967 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51002: callId: 704 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:34,009 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:34,009 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51002: callId: 706 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:34,039 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=48, retries=350, started=23591 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:34,110 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:34,110 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51002: callId: 708 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:34,213 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:34,213 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51002: callId: 710 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:34,240 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=49, retries=350, started=23792 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:34,319 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:34,319 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51002: callId: 712 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:34,421 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:34,421 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51002: callId: 714 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:34,421 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=501 ms ago, cancelled=false, msg= 2015-06-17 11:07:34,441 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=50, retries=350, started=23993 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:34,623 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:34,624 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51002: callId: 716 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:34,624 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=704 ms ago, cancelled=false, msg= 2015-06-17 11:07:34,641 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=51, retries=350, started=24193 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:34,826 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:34,826 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51002: callId: 718 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:34,827 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=907 ms ago, cancelled=false, msg= 2015-06-17 11:07:34,841 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=52, retries=350, started=24393 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:35,029 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:35,029 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51002: callId: 720 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:35,029 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1109 ms ago, cancelled=false, msg= 2015-06-17 11:07:35,044 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=53, retries=350, started=24596 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:35,231 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:35,232 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51002: callId: 722 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:35,232 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1312 ms ago, cancelled=false, msg= 2015-06-17 11:07:35,244 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=54, retries=350, started=24796 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:35,434 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:35,434 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51002: callId: 724 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:35,434 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1514 ms ago, cancelled=false, msg= 2015-06-17 11:07:35,445 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=55, retries=350, started=24997 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:35,636 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:35,636 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51002: callId: 726 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:35,637 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1717 ms ago, cancelled=false, msg= 2015-06-17 11:07:35,648 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=56, retries=350, started=25200 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:35,839 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:35,839 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51002: callId: 728 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:35,839 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1919 ms ago, cancelled=false, msg= 2015-06-17 11:07:35,850 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=57, retries=350, started=25402 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:36,042 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:36,042 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51002: callId: 730 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:36,042 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2122 ms ago, cancelled=false, msg= 2015-06-17 11:07:36,051 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=58, retries=350, started=25603 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:36,245 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:36,245 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51002: callId: 732 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:36,246 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2326 ms ago, cancelled=false, msg= 2015-06-17 11:07:36,253 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=59, retries=350, started=25805 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:36,449 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:36,449 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51002: callId: 734 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:36,449 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2529 ms ago, cancelled=false, msg= 2015-06-17 11:07:36,454 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=60, retries=350, started=26006 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:36,651 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:36,651 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51002: callId: 736 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:36,651 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2731 ms ago, cancelled=false, msg= 2015-06-17 11:07:36,654 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=61, retries=350, started=26206 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:36,827 DEBUG [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(188): Sleeping for 5468 before executing next master restart count=3 2015-06-17 11:07:36,854 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:36,854 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51002: callId: 738 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:36,854 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2934 ms ago, cancelled=false, msg= 2015-06-17 11:07:36,855 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=62, retries=350, started=26407 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:37,056 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=63, retries=350, started=26608 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:37,057 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:37,057 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51002: callId: 740 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:37,057 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3137 ms ago, cancelled=false, msg= 2015-06-17 11:07:37,256 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=64, retries=350, started=26808 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:37,260 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:37,260 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51002: callId: 742 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:37,260 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3340 ms ago, cancelled=false, msg= 2015-06-17 11:07:37,457 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=65, retries=350, started=27009 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:37,463 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:37,463 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51002: callId: 744 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:37,463 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3543 ms ago, cancelled=false, msg= 2015-06-17 11:07:37,657 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=66, retries=350, started=27209 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:37,665 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:37,666 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51002: callId: 746 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:37,666 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3746 ms ago, cancelled=false, msg= 2015-06-17 11:07:37,858 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=67, retries=350, started=27410 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:37,869 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:37,869 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51002: callId: 748 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:37,869 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3949 ms ago, cancelled=false, msg= 2015-06-17 11:07:38,059 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=68, retries=350, started=27611 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:38,071 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:38,071 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51002: callId: 750 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:38,072 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4152 ms ago, cancelled=false, msg= 2015-06-17 11:07:38,260 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=69, retries=350, started=27812 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:38,274 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:38,274 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51002: callId: 752 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:38,275 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4355 ms ago, cancelled=false, msg= 2015-06-17 11:07:38,461 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=70, retries=350, started=28013 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:38,477 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:38,477 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51002: callId: 754 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:38,477 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4557 ms ago, cancelled=false, msg= 2015-06-17 11:07:38,662 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=71, retries=350, started=28214 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:38,679 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:38,679 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51002: callId: 756 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:38,679 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4759 ms ago, cancelled=false, msg= 2015-06-17 11:07:38,863 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=72, retries=350, started=28415 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:38,881 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:38,882 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51002: callId: 758 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:38,882 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4962 ms ago, cancelled=false, msg= 2015-06-17 11:07:39,063 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=73, retries=350, started=28615 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:39,085 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:39,085 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51002: callId: 760 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:39,086 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5166 ms ago, cancelled=false, msg= 2015-06-17 11:07:39,266 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=74, retries=350, started=28818 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:39,289 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:39,289 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51002: callId: 762 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:39,289 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5369 ms ago, cancelled=false, msg= 2015-06-17 11:07:39,290 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:07:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564453920, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:07:39,467 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=75, retries=350, started=29019 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:39,492 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:39,492 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51002: callId: 764 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:39,495 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:39,495 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51002: callId: 766 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:39,499 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:39,499 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51002: callId: 768 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:39,505 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:39,505 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51002: callId: 770 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:39,517 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:39,517 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51002: callId: 772 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:39,538 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:39,539 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51002: callId: 774 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:39,580 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:39,580 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51002: callId: 776 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:39,668 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=76, retries=350, started=29220 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:39,681 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:39,681 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51002: callId: 778 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:39,784 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:39,784 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51002: callId: 780 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:39,869 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=77, retries=350, started=29421 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:39,885 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:39,886 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51002: callId: 782 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:39,987 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:39,987 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51002: callId: 784 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:39,987 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=496 ms ago, cancelled=false, msg= 2015-06-17 11:07:40,070 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=78, retries=350, started=29622 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:40,189 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:40,189 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51002: callId: 786 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:40,190 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=699 ms ago, cancelled=false, msg= 2015-06-17 11:07:40,272 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=79, retries=350, started=29824 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:40,391 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:40,391 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51002: callId: 788 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:40,392 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=901 ms ago, cancelled=false, msg= 2015-06-17 11:07:40,473 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=80, retries=350, started=30025 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:40,594 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:40,594 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51002: callId: 790 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:40,594 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1103 ms ago, cancelled=false, msg= 2015-06-17 11:07:40,676 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=81, retries=350, started=30227 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:40,797 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:40,798 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51002: callId: 792 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:40,798 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1307 ms ago, cancelled=false, msg= 2015-06-17 11:07:40,877 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=82, retries=350, started=30429 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:41,001 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:41,001 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51002: callId: 794 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:41,002 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1511 ms ago, cancelled=false, msg= 2015-06-17 11:07:41,078 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=83, retries=350, started=30630 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:41,204 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:41,204 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51002: callId: 796 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:41,204 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1713 ms ago, cancelled=false, msg= 2015-06-17 11:07:41,279 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=84, retries=350, started=30831 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:41,406 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:41,406 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51002: callId: 798 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:41,407 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1916 ms ago, cancelled=false, msg= 2015-06-17 11:07:41,481 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=85, retries=350, started=31033 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:41,609 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:41,609 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51002: callId: 800 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:41,610 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2119 ms ago, cancelled=false, msg= 2015-06-17 11:07:41,683 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=86, retries=350, started=31234 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:41,811 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:41,811 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51002: callId: 802 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:41,811 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2320 ms ago, cancelled=false, msg= 2015-06-17 11:07:41,885 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=87, retries=350, started=31437 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:42,012 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:42,012 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51002: callId: 804 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:42,013 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2522 ms ago, cancelled=false, msg= 2015-06-17 11:07:42,085 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=88, retries=350, started=31637 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:42,214 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:42,214 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51002: callId: 806 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:42,215 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2724 ms ago, cancelled=false, msg= 2015-06-17 11:07:42,285 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=89, retries=350, started=31837 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:42,295 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(197): executing master kill 2015-06-17 11:07:42,295 INFO [MasterRestartThread] hbase.MiniHBaseCluster(422): Aborting 10.0.0.110,51002,1434564446765 2015-06-17 11:07:42,295 FATAL [MasterRestartThread] master.HMaster(1953): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2015-06-17 11:07:42,295 FATAL [MasterRestartThread] master.HMaster(1956): Aborting for tests java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:423) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:270) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$5.run(TestMasterRestartWithProcedures.java:198) 2015-06-17 11:07:42,295 INFO [MasterRestartThread] regionserver.HRegionServer(1859): STOPPED: Aborting for tests 2015-06-17 11:07:42,295 INFO [MasterRestartThread] hbase.LocalHBaseCluster(379): Waiting on 10.0.0.110,51002,1434564446765 2015-06-17 11:07:42,295 INFO [M:0;10.0.0.110:51002] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2015-06-17 11:07:42,295 INFO [M:0;10.0.0.110:51002] regionserver.HeapMemoryManager(201): Stoping HeapMemoryTuner chore. 2015-06-17 11:07:42,295 INFO [SplitLogWorker-10.0.0.110:51002] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2015-06-17 11:07:42,295 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2015-06-17 11:07:42,296 INFO [M:0;10.0.0.110:51002] procedure2.ProcedureExecutor(490): Stopping the procedure executor 2015-06-17 11:07:42,296 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2015-06-17 11:07:42,296 INFO [SplitLogWorker-10.0.0.110:51002] regionserver.SplitLogWorker(155): SplitLogWorker 10.0.0.110,51002,1434564446765 exiting 2015-06-17 11:07:42,297 INFO [M:0;10.0.0.110:51002] wal.WALProcedureStore(159): Stopping the WAL Procedure Store 2015-06-17 11:07:42,303 INFO [IPC Server handler 6 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741865_1041{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 60 2015-06-17 11:07:42,379 INFO [ReplicationExecutor-0] replication.ReplicationQueuesZKImpl(171): Atomically moving 10.0.0.110,50903,1434564404489's WALs to my queue 2015-06-17 11:07:42,419 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:42,419 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51002: callId: 808 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:42,420 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2929 ms ago, cancelled=false, msg= 2015-06-17 11:07:42,488 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=90, retries=350, started=32039 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:42,624 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:42,624 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51002: callId: 810 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:42,625 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3134 ms ago, cancelled=false, msg= 2015-06-17 11:07:42,690 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=91, retries=350, started=32242 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:42,705 INFO [M:0;10.0.0.110:51002] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2015-06-17 11:07:42,706 INFO [M:0;10.0.0.110:51002] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2015-06-17 11:07:42,706 INFO [M:0;10.0.0.110:51002] regionserver.HRegionServer(1021): stopping server 10.0.0.110,51002,1434564446765 2015-06-17 11:07:42,706 DEBUG [M:0;10.0.0.110:51002] zookeeper.MetaTableLocator(617): Stopping MetaTableLocator 2015-06-17 11:07:42,706 INFO [M:0;10.0.0.110:51002] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320014 2015-06-17 11:07:42,708 DEBUG [M:0;10.0.0.110:51002] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:07:42,709 INFO [M:0;10.0.0.110:51002] regionserver.HRegionServer(1049): stopping server 10.0.0.110,51002,1434564446765; all regions closed. 2015-06-17 11:07:42,709 DEBUG [M:0;10.0.0.110:51002] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51002,1434564446765 2015-06-17 11:07:42,709 DEBUG [AsyncRpcChannel-pool2-t13] ipc.AsyncRpcChannel$8(574): IPC Client (-1632924467) connection to /10.0.0.110:50975 from ndimiduk.hfs.3: closed 2015-06-17 11:07:42,710 INFO [10.0.0.110:51002.activeMasterManager] zookeeper.MetaTableLocator(340): Failed verification of hbase:meta,,1 at address=10.0.0.110,50975,1434564429984, exception=Call id=0 on server /10.0.0.110:50975 aborted: connection is closing 2015-06-17 11:07:42,710 DEBUG [10.0.0.110:51002.activeMasterManager] zookeeper.ZKUtil(745): master:51002-0x14e02b3a2320013, quorum=localhost:51848, baseZNode=/hbase Unable to get data of znode /hbase/recovering-regions/1588230740 because node does not exist (not an error) 2015-06-17 11:07:42,712 INFO [IPC Server handler 8 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741864_1040{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 91 2015-06-17 11:07:42,713 DEBUG [10.0.0.110:51002.activeMasterManager] coordination.ZKSplitLogManagerCoordination(650): Marked 1588230740 recovering from 10.0.0.110,50975,1434564429984: /hbase/recovering-regions/1588230740/10.0.0.110,50975,1434564429984 2015-06-17 11:07:42,713 INFO [10.0.0.110:51002.activeMasterManager] master.HMaster(884): Re-assigning hbase:meta with replicaId, 0 it was on 10.0.0.110,50975,1434564429984 2015-06-17 11:07:42,713 INFO [10.0.0.110:51002.activeMasterManager] master.RegionStates(1072): Transition {1588230740 state=OPEN, ts=1434564446903, server=10.0.0.110,50975,1434564429984} to {1588230740 state=OFFLINE, ts=1434564462713, server=10.0.0.110,50975,1434564429984} 2015-06-17 11:07:42,713 INFO [10.0.0.110:51002.activeMasterManager] zookeeper.MetaTableLocator(440): Setting hbase:meta region location in ZooKeeper as 10.0.0.110,50975,1434564429984 2015-06-17 11:07:42,714 INFO [10.0.0.110:51002.activeMasterManager] master.AssignmentManager(996): Skip assigning hbase:meta,,1.1588230740, the server is stopped/aborted 2015-06-17 11:07:42,714 FATAL [10.0.0.110:51002.activeMasterManager] master.HMaster$1(1546): Failed to become active master java.io.IOException: connection is closed at org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:254) at org.apache.hadoop.hbase.MetaTableAccessor.putToMetaTable(MetaTableAccessor.java:1341) at org.apache.hadoop.hbase.MetaTableAccessor.updateTableState(MetaTableAccessor.java:1691) at org.apache.hadoop.hbase.MetaTableAccessor.updateTableState(MetaTableAccessor.java:1110) at org.apache.hadoop.hbase.master.TableStateManager.udpateMetaState(TableStateManager.java:180) at org.apache.hadoop.hbase.master.TableStateManager.setTableState(TableStateManager.java:67) at org.apache.hadoop.hbase.master.HMaster.assignMeta(HMaster.java:892) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:711) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1542) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:42,806 INFO [10.0.0.110,51002,1434564446765_splitLogManager__ChoreService_1] hbase.ScheduledChore(179): Chore: SplitLogManager Timeout Monitor was stopped 2015-06-17 11:07:42,829 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:42,830 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51002: callId: 812 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:42,830 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3339 ms ago, cancelled=false, msg= 2015-06-17 11:07:42,865 INFO [10.0.0.110,51002,1434564446765_ChoreService_1] hbase.ScheduledChore(179): Chore: 10.0.0.110,51002,1434564446765-MemstoreFlusherChore was stopped 2015-06-17 11:07:42,872 INFO [master//10.0.0.110:0.logRoller] regionserver.LogRoller(159): LogRoller exiting. 2015-06-17 11:07:42,872 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(146): master//10.0.0.110:0.leaseChecker closing leases 2015-06-17 11:07:42,873 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(149): master//10.0.0.110:0.leaseChecker closed leases 2015-06-17 11:07:42,892 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=92, retries=350, started=32444 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:43,035 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:43,035 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51002] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51002: callId: 814 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51011 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:43,036 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3545 ms ago, cancelled=false, msg= 2015-06-17 11:07:43,094 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=93, retries=350, started=32646 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:43,120 DEBUG [M:0;10.0.0.110:51002] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:07:43,120 INFO [M:0;10.0.0.110:51002] wal.FSHLog(977): Closed WAL: FSHLog 10.0.0.110%2C51002%2C1434564446765.default:(num 1434564446855) 2015-06-17 11:07:43,120 DEBUG [M:0;10.0.0.110:51002] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:07:43,121 INFO [M:0;10.0.0.110:51002] regionserver.Leases(146): M:0;10.0.0.110:51002 closing leases 2015-06-17 11:07:43,121 INFO [M:0;10.0.0.110:51002] regionserver.Leases(149): M:0;10.0.0.110:51002 closed leases 2015-06-17 11:07:43,121 INFO [M:0;10.0.0.110:51002] hbase.ChoreService(302): Chore service for: 10.0.0.110,51002,1434564446765 had [[ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region 10.0.0.110,51002,1434564446765 Period: 120000 Unit: MILLISECONDS]] on shutdown 2015-06-17 11:07:43,121 INFO [M:0;10.0.0.110:51002] regionserver.CompactSplitThread(394): Waiting for Split Thread to finish... 2015-06-17 11:07:43,121 INFO [M:0;10.0.0.110:51002] regionserver.CompactSplitThread(394): Waiting for Merge Thread to finish... 2015-06-17 11:07:43,121 INFO [M:0;10.0.0.110:51002] regionserver.CompactSplitThread(394): Waiting for Large Compaction Thread to finish... 2015-06-17 11:07:43,121 INFO [M:0;10.0.0.110:51002] regionserver.CompactSplitThread(394): Waiting for Small Compaction Thread to finish... 2015-06-17 11:07:43,123 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51002-0x14e02b3a2320013, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/10.0.0.110,51002,1434564446765 2015-06-17 11:07:43,123 INFO [M:0;10.0.0.110:51002] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320017 2015-06-17 11:07:43,124 DEBUG [M:0;10.0.0.110:51002] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:07:43,124 DEBUG [M:0;10.0.0.110:51002] master.HMaster(1074): Stopping service threads 2015-06-17 11:07:43,125 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51002-0x14e02b3a2320013, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:07:43,125 INFO [M:0;10.0.0.110:51002] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320015 2015-06-17 11:07:43,125 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:07:43,126 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(488): master:51002-0x14e02b3a2320013, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:07:43,126 DEBUG [M:0;10.0.0.110:51002] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:07:43,126 DEBUG [main-EventThread] zookeeper.ZKUtil(488): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:07:43,126 INFO [M:0;10.0.0.110:51002] hbase.ChoreService(302): Chore service for: 10.0.0.110,51002,1434564446765_splitLogManager_ had [] on shutdown 2015-06-17 11:07:43,126 INFO [M:0;10.0.0.110:51002] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2015-06-17 11:07:43,126 INFO [M:0;10.0.0.110:51002] ipc.RpcServer(2219): Stopping server on 51002 2015-06-17 11:07:43,126 INFO [RpcServer.listener,port=51002] ipc.RpcServer$Listener(776): RpcServer.listener,port=51002: stopping 2015-06-17 11:07:43,127 INFO [RpcServer.responder] ipc.RpcServer$Responder(1018): RpcServer.responder: stopped 2015-06-17 11:07:43,127 INFO [RpcServer.responder] ipc.RpcServer$Responder(921): RpcServer.responder: stopping 2015-06-17 11:07:43,128 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51002-0x14e02b3a2320013, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,51002,1434564446765 2015-06-17 11:07:43,128 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,51002,1434564446765 2015-06-17 11:07:43,128 INFO [MasterRestartThread-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [10.0.0.110,51002,1434564446765] 2015-06-17 11:07:43,128 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51002-0x14e02b3a2320013, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:07:43,128 INFO [M:0;10.0.0.110:51002] regionserver.HRegionServer(1093): stopping server 10.0.0.110,51002,1434564446765; zookeeper connection closed. 2015-06-17 11:07:43,129 INFO [M:0;10.0.0.110:51002] regionserver.HRegionServer(1096): M:0;10.0.0.110:51002 exiting 2015-06-17 11:07:43,129 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(200): executing master restart 2015-06-17 11:07:43,129 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:43,129 INFO [main-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(121): /hbase/rs/10.0.0.110,51002,1434564446765 znode expired, triggering replicatorRemoved event 2015-06-17 11:07:43,129 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:07:43,130 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:43,142 INFO [MasterRestartThread] client.ConnectionUtils(105): master//10.0.0.110:0 server-side HConnection retries=350 2015-06-17 11:07:43,143 INFO [MasterRestartThread] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2015-06-17 11:07:43,166 INFO [MasterRestartThread] ipc.RpcServer$Listener(598): master//10.0.0.110:0: started 10 reader(s). 2015-06-17 11:07:43,168 INFO [MasterRestartThread] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:07:43,169 INFO [MasterRestartThread] zookeeper.RecoverableZooKeeper(121): Process identifier=master:51015 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:07:43,171 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:510150x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:07:43,172 DEBUG [MasterRestartThread] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-17 11:07:43,173 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(578): master:51015-0x14e02b3a2320018 connected 2015-06-17 11:07:43,174 DEBUG [MasterRestartThread] zookeeper.ZKUtil(488): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:07:43,175 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2015-06-17 11:07:43,175 INFO [RpcServer.responder] ipc.RpcServer$Responder(917): RpcServer.responder: starting 2015-06-17 11:07:43,175 INFO [RpcServer.listener,port=51015] ipc.RpcServer$Listener(729): RpcServer.listener,port=51015: starting 2015-06-17 11:07:43,175 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=0 queue=0 2015-06-17 11:07:43,176 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=1 queue=0 2015-06-17 11:07:43,176 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=2 queue=0 2015-06-17 11:07:43,176 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=3 queue=0 2015-06-17 11:07:43,176 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=4 queue=0 2015-06-17 11:07:43,176 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=0 queue=0 2015-06-17 11:07:43,177 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=1 queue=1 2015-06-17 11:07:43,177 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=2 queue=0 2015-06-17 11:07:43,177 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=3 queue=1 2015-06-17 11:07:43,177 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=4 queue=0 2015-06-17 11:07:43,177 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=5 queue=1 2015-06-17 11:07:43,177 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=6 queue=0 2015-06-17 11:07:43,177 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=7 queue=1 2015-06-17 11:07:43,177 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=8 queue=0 2015-06-17 11:07:43,177 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=9 queue=1 2015-06-17 11:07:43,178 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=10 queue=0 2015-06-17 11:07:43,178 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=11 queue=1 2015-06-17 11:07:43,178 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=12 queue=0 2015-06-17 11:07:43,178 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=13 queue=1 2015-06-17 11:07:43,178 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=14 queue=0 2015-06-17 11:07:43,178 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=15 queue=1 2015-06-17 11:07:43,178 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=16 queue=0 2015-06-17 11:07:43,178 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=17 queue=1 2015-06-17 11:07:43,179 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=18 queue=0 2015-06-17 11:07:43,179 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=19 queue=1 2015-06-17 11:07:43,179 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=0 queue=0 2015-06-17 11:07:43,179 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=1 queue=0 2015-06-17 11:07:43,179 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=2 queue=0 2015-06-17 11:07:43,179 INFO [MasterRestartThread] master.HMaster(349): hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034, hbase.cluster.distributed=false 2015-06-17 11:07:43,179 INFO [MasterRestartThread] master.HMaster(1512): Adding backup master ZNode /hbase/backup-masters/10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,180 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,182 INFO [M:0;10.0.0.110:51015] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x62523e8d connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:07:43,182 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:07:43,182 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:07:43,183 DEBUG [10.0.0.110:51015.activeMasterManager] zookeeper.ZKUtil(486): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:07:43,183 INFO [10.0.0.110:51015.activeMasterManager] master.ActiveMasterManager(170): Deleting ZNode for /hbase/backup-masters/10.0.0.110,51015,1434564463167 from backup master directory 2015-06-17 11:07:43,183 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(486): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:07:43,183 DEBUG [MasterRestartThread-EventThread] master.ActiveMasterManager(126): A master is now available 2015-06-17 11:07:43,184 DEBUG [M:0;10.0.0.110:51015-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x62523e8d0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:07:43,184 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,184 WARN [10.0.0.110:51015.activeMasterManager] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:07:43,184 INFO [10.0.0.110:51015.activeMasterManager] master.ActiveMasterManager(179): Registered Active Master=10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,185 DEBUG [M:0;10.0.0.110:51015] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@579b2abd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:07:43,185 DEBUG [M:0;10.0.0.110:51015] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:07:43,185 DEBUG [M:0;10.0.0.110:51015] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:07:43,185 DEBUG [M:0;10.0.0.110:51015-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x62523e8d-0x14e02b3a2320019 connected 2015-06-17 11:07:43,186 INFO [M:0;10.0.0.110:51015] regionserver.HRegionServer(774): ClusterId : b7c4b21b-aab2-44b4-8562-2b61af22ad6c 2015-06-17 11:07:43,186 DEBUG [M:0;10.0.0.110:51015] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2015-06-17 11:07:43,187 DEBUG [M:0;10.0.0.110:51015] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:07:43,187 DEBUG [M:0;10.0.0.110:51015] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2015-06-17 11:07:43,187 DEBUG [M:0;10.0.0.110:51015] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2015-06-17 11:07:43,188 DEBUG [M:0;10.0.0.110:51015] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:07:43,189 DEBUG [M:0;10.0.0.110:51015] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2015-06-17 11:07:43,189 INFO [M:0;10.0.0.110:51015] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2015-06-17 11:07:43,189 INFO [M:0;10.0.0.110:51015] compactions.PressureAwareCompactionThroughputController(175): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2015-06-17 11:07:43,189 INFO [M:0;10.0.0.110:51015] regionserver.HRegionServer$CompactionChecker(1504): CompactionChecker runs every 1sec 2015-06-17 11:07:43,190 DEBUG [M:0;10.0.0.110:51015] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fe4351d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=10.0.0.110/10.0.0.110:0 2015-06-17 11:07:43,190 DEBUG [M:0;10.0.0.110:51015] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:07:43,190 DEBUG [M:0;10.0.0.110:51015] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:07:43,190 DEBUG [M:0;10.0.0.110:51015] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:M:0;10.0.0.110:51015 2015-06-17 11:07:43,191 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:07:43,191 DEBUG [M:0;10.0.0.110:51015] zookeeper.ZKUtil(486): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,191 INFO [M:0;10.0.0.110:51015] regionserver.RegionServerCoprocessorHost(65): System coprocessor loading is enabled 2015-06-17 11:07:43,191 INFO [M:0;10.0.0.110:51015] regionserver.RegionServerCoprocessorHost(66): Table coprocessor loading is enabled 2015-06-17 11:07:43,191 INFO [M:0;10.0.0.110:51015] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,51015,1434564463167 with port=51015, startcode=1434564463167 2015-06-17 11:07:43,191 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:43,191 DEBUG [M:0;10.0.0.110:51015] regionserver.HRegionServer(2295): Master is not running yet 2015-06-17 11:07:43,191 WARN [M:0;10.0.0.110:51015] regionserver.HRegionServer(900): reportForDuty failed; sleeping and then retrying. 2015-06-17 11:07:43,192 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,196 DEBUG [10.0.0.110:51015.activeMasterManager] util.FSTableDescriptors(807): Current tableInfoPath = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2015-06-17 11:07:43,199 DEBUG [10.0.0.110:51015.activeMasterManager] util.FSTableDescriptors(811): TableInfo already exists.. Skipping creation 2015-06-17 11:07:43,202 INFO [10.0.0.110:51015.activeMasterManager] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:07:43,202 DEBUG [10.0.0.110:51015.activeMasterManager] coordination.ZKSplitLogManagerCoordination(885): Distributed log replay=true 2015-06-17 11:07:43,203 INFO [10.0.0.110:51015.activeMasterManager] coordination.ZKSplitLogManagerCoordination(599): Found 0 orphan tasks and 0 rescan nodes 2015-06-17 11:07:43,203 DEBUG [10.0.0.110:51015.activeMasterManager] util.FSTableDescriptors(222): Fetching table descriptors from the filesystem. 2015-06-17 11:07:43,214 INFO [10.0.0.110:51015.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x1ee6841 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:07:43,216 DEBUG [10.0.0.110:51015.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x1ee68410x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:07:43,216 DEBUG [10.0.0.110:51015.activeMasterManager] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50c87dff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:07:43,216 DEBUG [10.0.0.110:51015.activeMasterManager] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:07:43,216 DEBUG [10.0.0.110:51015.activeMasterManager] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:07:43,217 DEBUG [10.0.0.110:51015.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x1ee6841-0x14e02b3a232001a connected 2015-06-17 11:07:43,217 INFO [10.0.0.110:51015.activeMasterManager] balancer.StochasticLoadBalancer(135): loading config 2015-06-17 11:07:43,218 DEBUG [10.0.0.110:51015.activeMasterManager] zookeeper.ZKUtil(488): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2015-06-17 11:07:43,218 DEBUG [10.0.0.110:51015.activeMasterManager] zookeeper.ZKUtil(486): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:43,219 DEBUG [10.0.0.110:51015.activeMasterManager] zookeeper.ZKUtil(486): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,219 DEBUG [10.0.0.110:51015.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:43,219 DEBUG [10.0.0.110:51015.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,219 INFO [10.0.0.110:51015.activeMasterManager] master.HMaster(568): Server active/primary master=10.0.0.110,51015,1434564463167, sessionid=0x14e02b3a2320018, setting cluster-up flag (Was=true) 2015-06-17 11:07:43,221 DEBUG [10.0.0.110:51015.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:07:43,221 INFO [10.0.0.110:51015.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2015-06-17 11:07:43,222 DEBUG [10.0.0.110:51015.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,223 DEBUG [10.0.0.110:51015.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:07:43,223 INFO [10.0.0.110:51015.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2015-06-17 11:07:43,224 DEBUG [10.0.0.110:51015.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,224 INFO [10.0.0.110:51015.activeMasterManager] master.MasterCoprocessorHost(86): System coprocessor loading is enabled 2015-06-17 11:07:43,224 DEBUG [10.0.0.110:51015.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_OPEN_REGION-10.0.0.110:51015, corePoolSize=5, maxPoolSize=5 2015-06-17 11:07:43,224 DEBUG [10.0.0.110:51015.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_CLOSE_REGION-10.0.0.110:51015, corePoolSize=5, maxPoolSize=5 2015-06-17 11:07:43,224 DEBUG [10.0.0.110:51015.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_SERVER_OPERATIONS-10.0.0.110:51015, corePoolSize=5, maxPoolSize=5 2015-06-17 11:07:43,224 DEBUG [10.0.0.110:51015.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_META_SERVER_OPERATIONS-10.0.0.110:51015, corePoolSize=5, maxPoolSize=5 2015-06-17 11:07:43,225 DEBUG [10.0.0.110:51015.activeMasterManager] executor.ExecutorService(91): Starting executor service name=M_LOG_REPLAY_OPS-10.0.0.110:51015, corePoolSize=10, maxPoolSize=10 2015-06-17 11:07:43,225 DEBUG [10.0.0.110:51015.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_TABLE_OPERATIONS-10.0.0.110:51015, corePoolSize=1, maxPoolSize=1 2015-06-17 11:07:43,225 INFO [10.0.0.110:51015.activeMasterManager] procedure2.ProcedureExecutor(446): Starting procedure executor threads=9 2015-06-17 11:07:43,225 INFO [10.0.0.110:51015.activeMasterManager] wal.WALProcedureStore(214): Starting WAL Procedure Store lease recovery 2015-06-17 11:07:43,226 INFO [10.0.0.110:51015.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log 2015-06-17 11:07:43,226 INFO [10.0.0.110:51015.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log after 0ms 2015-06-17 11:07:43,226 DEBUG [10.0.0.110:51015.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log; isDirectory=false; length=14915; replication=1; blocksize=134217728; modification_time=1434564413352; access_time=1434564405122; owner=ndimiduk; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:07:43,229 INFO [10.0.0.110:51015.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000004.log 2015-06-17 11:07:43,229 INFO [10.0.0.110:51015.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000004.log after 0ms 2015-06-17 11:07:43,229 DEBUG [10.0.0.110:51015.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000004.log; isDirectory=false; length=60; replication=1; blocksize=134217728; modification_time=1434564462705; access_time=1434564446835; owner=ndimiduk.hfs.3; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:07:43,233 DEBUG [10.0.0.110:51015.activeMasterManager] wal.WALProcedureStore(610): Roll new state log: 5 2015-06-17 11:07:43,234 INFO [10.0.0.110:51015.activeMasterManager] wal.WALProcedureStore(237): Lease acquired for flushLogId: 5 2015-06-17 11:07:43,235 INFO [10.0.0.110:51015.activeMasterManager] wal.ProcedureWALFormatReader(150): No active entry found in state log hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000004.log. removing it 2015-06-17 11:07:43,236 DEBUG [10.0.0.110:51015.activeMasterManager] procedure2.ProcedureExecutor$1(278): load procedures maxProcId=7 2015-06-17 11:07:43,237 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:07:43,237 DEBUG [10.0.0.110:51015.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=RUNNABLE isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=7 state=RUNNABLE 2015-06-17 11:07:43,238 DEBUG [10.0.0.110:51015.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DeleteTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=6 state=FINISHED 2015-06-17 11:07:43,238 DEBUG [10.0.0.110:51015.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:43,238 DEBUG [10.0.0.110:51015.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=5 state=FINISHED 2015-06-17 11:07:43,238 DEBUG [10.0.0.110:51015.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:43,238 DEBUG [10.0.0.110:51015.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: EnableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=4 state=FINISHED 2015-06-17 11:07:43,238 DEBUG [10.0.0.110:51015.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:43,238 DEBUG [10.0.0.110:51015.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=3 state=FINISHED 2015-06-17 11:07:43,238 DEBUG [10.0.0.110:51015.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:43,238 DEBUG [10.0.0.110:51015.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=2 state=FINISHED 2015-06-17 11:07:43,238 DEBUG [10.0.0.110:51015.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:43,239 DEBUG [10.0.0.110:51015.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=hbase:namespace) user=ndimiduk (auth:SIMPLE) id=1 state=FINISHED 2015-06-17 11:07:43,239 DEBUG [10.0.0.110:51015.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:43,239 DEBUG [AsyncRpcChannel-pool2-t16] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service MasterService, sasl=false 2015-06-17 11:07:43,239 DEBUG [RpcServer.listener,port=51015] ipc.RpcServer$Listener(840): RpcServer.listener,port=51015: connection from 10.0.0.110:51022; # active connections: 1 2015-06-17 11:07:43,239 DEBUG [10.0.0.110:51015.activeMasterManager] wal.WALProcedureStore(652): Remove log: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000004.log 2015-06-17 11:07:43,240 INFO [RpcServer.reader=1,bindAddress=10.0.0.110,port=51015] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51022 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:07:43,240 INFO [IPC Server handler 6 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741865_1041 127.0.0.1:50896 2015-06-17 11:07:43,240 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51015: callId: 816 service: MasterService methodName: IsMasterRunning size: 24 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:43,241 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3750 ms ago, cancelled=false, msg= 2015-06-17 11:07:43,241 DEBUG [10.0.0.110:51015.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2015-06-17 11:07:43,242 INFO [10.0.0.110:51015.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:07:43,244 DEBUG [10.0.0.110:51015.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:07:43,244 DEBUG [10.0.0.110:51015.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/replication/rs already exists 2015-06-17 11:07:43,244 DEBUG [10.0.0.110:51015.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2015-06-17 11:07:43,245 DEBUG [10.0.0.110:51015.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner 2015-06-17 11:07:43,245 DEBUG [10.0.0.110:51015.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): replicationLogCleaner-0x14e02b3a232001b connected 2015-06-17 11:07:43,245 DEBUG [10.0.0.110:51015.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2015-06-17 11:07:43,245 DEBUG [10.0.0.110:51015.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2015-06-17 11:07:43,246 DEBUG [10.0.0.110:51015.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2015-06-17 11:07:43,246 INFO [10.0.0.110:51015.activeMasterManager] master.ServerManager(1011): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 1, timeout of 4500 ms, interval of 1500 ms. 2015-06-17 11:07:43,246 INFO [M:0;10.0.0.110:51015] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,51015,1434564463167 with port=51015, startcode=1434564463167 2015-06-17 11:07:43,246 INFO [M:0;10.0.0.110:51015] master.ServerManager(459): Registering server=10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,246 INFO [M:0;10.0.0.110:51015] regionserver.HRegionServer(1341): Config from master: hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034 2015-06-17 11:07:43,246 INFO [M:0;10.0.0.110:51015] regionserver.HRegionServer(1341): Config from master: fs.defaultFS=hdfs://localhost:50895 2015-06-17 11:07:43,246 INFO [M:0;10.0.0.110:51015] regionserver.HRegionServer(1341): Config from master: hbase.master.info.port=-1 2015-06-17 11:07:43,246 WARN [M:0;10.0.0.110:51015] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:07:43,246 INFO [M:0;10.0.0.110:51015] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:07:43,246 DEBUG [M:0;10.0.0.110:51015] regionserver.HRegionServer(1603): logdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,249 DEBUG [M:0;10.0.0.110:51015] regionserver.Replication(142): ReplicationStatisticsThread 300 2015-06-17 11:07:43,249 INFO [M:0;10.0.0.110:51015] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:07:43,252 INFO [M:0;10.0.0.110:51015] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.0.0.110%2C51015%2C1434564463167.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51015,1434564463167, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:07:43,257 INFO [M:0;10.0.0.110:51015] wal.FSHLog(1365): Slow sync cost: 3 ms, current pipeline: [] 2015-06-17 11:07:43,258 INFO [M:0;10.0.0.110:51015] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51015,1434564463167/10.0.0.110%2C51015%2C1434564463167.default.1434564463252 2015-06-17 11:07:43,258 INFO [M:0;10.0.0.110:51015] regionserver.MetricsRegionServerWrapperImpl(113): Computing regionserver metrics every 5000 milliseconds 2015-06-17 11:07:43,259 DEBUG [M:0;10.0.0.110:51015] executor.ExecutorService(91): Starting executor service name=RS_OPEN_REGION-10.0.0.110:51015, corePoolSize=3, maxPoolSize=3 2015-06-17 11:07:43,259 DEBUG [M:0;10.0.0.110:51015] executor.ExecutorService(91): Starting executor service name=RS_OPEN_META-10.0.0.110:51015, corePoolSize=1, maxPoolSize=1 2015-06-17 11:07:43,259 DEBUG [M:0;10.0.0.110:51015] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_REGION-10.0.0.110:51015, corePoolSize=3, maxPoolSize=3 2015-06-17 11:07:43,259 DEBUG [M:0;10.0.0.110:51015] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_META-10.0.0.110:51015, corePoolSize=1, maxPoolSize=1 2015-06-17 11:07:43,259 DEBUG [M:0;10.0.0.110:51015] executor.ExecutorService(91): Starting executor service name=RS_LOG_REPLAY_OPS-10.0.0.110:51015, corePoolSize=2, maxPoolSize=2 2015-06-17 11:07:43,259 DEBUG [M:0;10.0.0.110:51015] executor.ExecutorService(91): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-10.0.0.110:51015, corePoolSize=3, maxPoolSize=3 2015-06-17 11:07:43,261 DEBUG [M:0;10.0.0.110:51015] zookeeper.ZKUtil(486): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:43,261 DEBUG [M:0;10.0.0.110:51015] zookeeper.ZKUtil(486): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,261 INFO [M:0;10.0.0.110:51015] regionserver.ReplicationSourceManager(229): Current list of replicators: [10.0.0.110,50905,1434564404764, 10.0.0.110,51015,1434564463167] other RSs: [10.0.0.110,50905,1434564404764, 10.0.0.110,51015,1434564463167] 2015-06-17 11:07:43,274 INFO [M:0;10.0.0.110:51015] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x56ccf7b8 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:07:43,276 DEBUG [M:0;10.0.0.110:51015-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x56ccf7b80x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:07:43,277 DEBUG [M:0;10.0.0.110:51015] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodecWithTags@5fe21ec9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:07:43,277 DEBUG [M:0;10.0.0.110:51015] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:07:43,277 DEBUG [M:0;10.0.0.110:51015] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:07:43,277 DEBUG [M:0;10.0.0.110:51015-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x56ccf7b8-0x14e02b3a232001c connected 2015-06-17 11:07:43,289 INFO [M:0;10.0.0.110:51015] regionserver.HeapMemoryManager(190): Starting HeapMemoryTuner chore. 2015-06-17 11:07:43,289 INFO [SplitLogWorker-10.0.0.110:51015] regionserver.SplitLogWorker(134): SplitLogWorker 10.0.0.110,51015,1434564463167 starting 2015-06-17 11:07:43,289 INFO [M:0;10.0.0.110:51015] regionserver.HRegionServer(1363): Serving as 10.0.0.110,51015,1434564463167, RpcServer on 10.0.0.110/10.0.0.110:51015, sessionid=0x14e02b3a2320018 2015-06-17 11:07:43,289 DEBUG [M:0;10.0.0.110:51015] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2015-06-17 11:07:43,289 DEBUG [M:0;10.0.0.110:51015] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager 10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,289 DEBUG [M:0;10.0.0.110:51015] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,51015,1434564463167' 2015-06-17 11:07:43,290 DEBUG [M:0;10.0.0.110:51015] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2015-06-17 11:07:43,290 DEBUG [M:0;10.0.0.110:51015] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2015-06-17 11:07:43,290 DEBUG [M:0;10.0.0.110:51015] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2015-06-17 11:07:43,290 DEBUG [M:0;10.0.0.110:51015] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2015-06-17 11:07:43,290 DEBUG [M:0;10.0.0.110:51015] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager 10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,290 DEBUG [M:0;10.0.0.110:51015] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,51015,1434564463167' 2015-06-17 11:07:43,290 DEBUG [M:0;10.0.0.110:51015] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2015-06-17 11:07:43,291 DEBUG [M:0;10.0.0.110:51015] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2015-06-17 11:07:43,291 DEBUG [M:0;10.0.0.110:51015] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2015-06-17 11:07:43,291 INFO [M:0;10.0.0.110:51015] quotas.RegionServerQuotaManager(62): Quota support disabled 2015-06-17 11:07:43,294 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=94, retries=350, started=32846 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:43,296 INFO [10.0.0.110:51015.activeMasterManager] master.ServerManager(1028): Finished waiting for region servers count to settle; checked in 1, slept for 50 ms, expecting minimum of 1, maximum of 1, master is running 2015-06-17 11:07:43,296 INFO [10.0.0.110:51015.activeMasterManager] master.ServerManager(459): Registering server=10.0.0.110,50905,1434564404764 2015-06-17 11:07:43,296 INFO [10.0.0.110:51015.activeMasterManager] master.HMaster(665): Registered server found up in zk but who has not yet reported in: 10.0.0.110,50905,1434564404764 2015-06-17 11:07:43,298 INFO [10.0.0.110:51015.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50905,1434564404764 belongs to an existing region server 2015-06-17 11:07:43,300 INFO [10.0.0.110:51015.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51015,1434564463167 belongs to an existing region server 2015-06-17 11:07:43,302 DEBUG [10.0.0.110:51015.activeMasterManager] coordination.ZKSplitLogManagerCoordination(650): Marked 1588230740 recovering from 10.0.0.110,50975,1434564429984: /hbase/recovering-regions/1588230740/10.0.0.110,50975,1434564429984 2015-06-17 11:07:43,302 INFO [10.0.0.110:51015.activeMasterManager] master.HMaster(884): Re-assigning hbase:meta with replicaId, 0 it was on 10.0.0.110,50975,1434564429984 2015-06-17 11:07:43,302 DEBUG [10.0.0.110:51015.activeMasterManager] master.AssignmentManager(1239): No previous transition plan found (or ignoring an existing plan) for hbase:meta,,1.1588230740; generated random plan=hri=hbase:meta,,1.1588230740, src=, dest=10.0.0.110,51015,1434564463167; 2 (online=2) available servers, forceNewPlan=false 2015-06-17 11:07:43,303 INFO [10.0.0.110:51015.activeMasterManager] master.AssignmentManager(1028): Assigning hbase:meta,,1.1588230740 to 10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,303 INFO [10.0.0.110:51015.activeMasterManager] master.RegionStates(1072): Transition {1588230740 state=OFFLINE, ts=1434564463302, server=10.0.0.110,50975,1434564429984} to {1588230740 state=PENDING_OPEN, ts=1434564463303, server=10.0.0.110,51015,1434564463167} 2015-06-17 11:07:43,303 INFO [10.0.0.110:51015.activeMasterManager] zookeeper.MetaTableLocator(440): Setting hbase:meta region location in ZooKeeper as 10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,303 DEBUG [10.0.0.110:51015.activeMasterManager] master.ServerManager(937): New admin connection to 10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,303 INFO [10.0.0.110:51015.activeMasterManager] regionserver.RSRpcServices(1432): Open hbase:meta,,1.1588230740 2015-06-17 11:07:43,304 INFO [RS_OPEN_META-10.0.0.110:51015-0] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:07:43,304 DEBUG [10.0.0.110:51015.activeMasterManager] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564463304,"tag":[],"qualifier":"state","vlen":2}]},"row":"hbase:meta"} 2015-06-17 11:07:43,306 INFO [RS_OPEN_META-10.0.0.110:51015-0] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.0.0.110%2C51015%2C1434564463167..meta, suffix=.meta, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51015,1434564463167, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:07:43,311 INFO [RS_OPEN_META-10.0.0.110:51015-0] wal.FSHLog(1365): Slow sync cost: 3 ms, current pipeline: [] 2015-06-17 11:07:43,311 INFO [RS_OPEN_META-10.0.0.110:51015-0] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51015,1434564463167/10.0.0.110%2C51015%2C1434564463167..meta.1434564463306.meta 2015-06-17 11:07:43,312 DEBUG [RS_OPEN_META-10.0.0.110:51015-0] regionserver.HRegion(6070): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2015-06-17 11:07:43,313 DEBUG [RS_OPEN_META-10.0.0.110:51015-0] coprocessor.CoprocessorHost(179): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2015-06-17 11:07:43,313 DEBUG [RS_OPEN_META-10.0.0.110:51015-0] regionserver.HRegion(7284): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2015-06-17 11:07:43,313 INFO [RS_OPEN_META-10.0.0.110:51015-0] regionserver.RegionCoprocessorHost(350): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2015-06-17 11:07:43,313 DEBUG [RS_OPEN_META-10.0.0.110:51015-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table meta 1588230740 2015-06-17 11:07:43,314 DEBUG [RS_OPEN_META-10.0.0.110:51015-0] regionserver.HRegion(722): Instantiated hbase:meta,,1.1588230740 2015-06-17 11:07:43,317 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:07:43,317 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:07:43,327 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/7acd1d3a04d44ab0a2c087c8d9edde78, isReference=false, isBulkLoadResult=false, seqid=40000117, majorCompaction=false 2015-06-17 11:07:43,334 INFO [StoreFileOpenerThread-info-1] regionserver.StoreFile$Reader(1439): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fd7dd8c37dcd4f58adb7e614dd55c26b 2015-06-17 11:07:43,334 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/fd7dd8c37dcd4f58adb7e614dd55c26b, isReference=false, isBulkLoadResult=false, seqid=101, majorCompaction=false 2015-06-17 11:07:43,335 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:07:43,335 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:07:43,345 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/329b922a4c424a6b9d97b18b74f7f17e, isReference=false, isBulkLoadResult=false, seqid=40000117, majorCompaction=false 2015-06-17 11:07:43,352 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/adda056a1f284c47be8d8b15ee043a41, isReference=false, isBulkLoadResult=false, seqid=101, majorCompaction=false 2015-06-17 11:07:43,354 DEBUG [RS_OPEN_META-10.0.0.110:51015-0] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740 2015-06-17 11:07:43,356 DEBUG [RS_OPEN_META-10.0.0.110:51015-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:07:43,359 DEBUG [RS_OPEN_META-10.0.0.110:51015-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/recovered.edits/80000120.seqid to file, newSeqId=80000120, maxSeqId=40000120 2015-06-17 11:07:43,360 INFO [RS_OPEN_META-10.0.0.110:51015-0] regionserver.HRegion(848): Onlined 1588230740; next sequenceid=80000120 2015-06-17 11:07:43,361 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1893): Post open deploy tasks for hbase:meta,,1.1588230740 2015-06-17 11:07:43,362 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/recovering-regions/1588230740 2015-06-17 11:07:43,363 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(3181): Update last flushed sequence id of region 1588230740 for 10.0.0.110,50975,1434564429984 2015-06-17 11:07:43,364 DEBUG [PostOpenDeployTasks:1588230740] master.AssignmentManager(2812): Got transition OPENED for {1588230740 state=PENDING_OPEN, ts=1434564463303, server=10.0.0.110,51015,1434564463167} from 10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,364 INFO [PostOpenDeployTasks:1588230740] master.RegionStates(1072): Transition {1588230740 state=PENDING_OPEN, ts=1434564463303, server=10.0.0.110,51015,1434564463167} to {1588230740 state=OPEN, ts=1434564463364, server=10.0.0.110,51015,1434564463167} 2015-06-17 11:07:43,364 INFO [PostOpenDeployTasks:1588230740] zookeeper.MetaTableLocator(440): Setting hbase:meta region location in ZooKeeper as 10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,364 DEBUG [PostOpenDeployTasks:1588230740] master.RegionStates(417): Onlined 1588230740 on 10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,365 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1920): Finished post open deploy task for hbase:meta,,1.1588230740 2015-06-17 11:07:43,365 DEBUG [RS_OPEN_META-10.0.0.110:51015-0] handler.OpenRegionHandler(125): Opened hbase:meta,,1.1588230740 on 10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,443 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:07:43,446 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:43,446 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51015: callId: 819 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:43,446 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3955 ms ago, cancelled=false, msg= 2015-06-17 11:07:43,495 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=95, retries=350, started=33047 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:43,508 DEBUG [AsyncRpcChannel-pool2-t1] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service ClientService, sasl=false 2015-06-17 11:07:43,508 DEBUG [RpcServer.listener,port=51015] ipc.RpcServer$Listener(840): RpcServer.listener,port=51015: connection from 10.0.0.110:51029; # active connections: 2 2015-06-17 11:07:43,509 INFO [RpcServer.reader=2,bindAddress=10.0.0.110,port=51015] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51029 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:07:43,511 INFO [10.0.0.110:51015.activeMasterManager] hbase.MetaTableAccessor(1692): Updated table hbase:meta state to ENABLED in META 2015-06-17 11:07:43,513 DEBUG [10.0.0.110:51015.activeMasterManager] master.MasterFileSystem(339): Renamed region directory: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50975,1434564429984-splitting 2015-06-17 11:07:43,513 INFO [10.0.0.110:51015.activeMasterManager] master.SplitLogManager(560): dead splitlog workers [10.0.0.110,50975,1434564429984] 2015-06-17 11:07:43,514 INFO [10.0.0.110:51015.activeMasterManager] master.SplitLogManager(192): hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50975,1434564429984-splitting is empty dir, no logs to split 2015-06-17 11:07:43,514 DEBUG [10.0.0.110:51015.activeMasterManager] master.SplitLogManager(255): Scheduling batch of logs to split 2015-06-17 11:07:43,514 INFO [10.0.0.110:51015.activeMasterManager] master.SplitLogManager(257): started splitting 0 logs in [hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50975,1434564429984-splitting] for [10.0.0.110,50975,1434564429984] 2015-06-17 11:07:43,515 DEBUG [10.0.0.110:51015.activeMasterManager] coordination.ZKSplitLogManagerCoordination(324): Processing recovering [1588230740] and servers [10.0.0.110,50975,1434564429984], isMetaRecovery=true 2015-06-17 11:07:43,516 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/recovering-regions/1588230740/10.0.0.110,50975,1434564429984 2015-06-17 11:07:43,517 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/recovering-regions/1588230740 2015-06-17 11:07:43,519 INFO [RS_OPEN_REGION-10.0.0.110:51015-0] handler.FinishRegionRecoveringHandler(52): /hbase/recovering-regions/1588230740 deleted; 1588230740 recovered. 2015-06-17 11:07:43,519 INFO [10.0.0.110:51015.activeMasterManager] master.SplitLogManager(316): finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50975,1434564429984-splitting] in 5ms 2015-06-17 11:07:43,519 DEBUG [10.0.0.110:51015.activeMasterManager] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564463519,"tag":[],"qualifier":"state","vlen":2}]},"row":"hbase:meta"} 2015-06-17 11:07:43,522 INFO [10.0.0.110:51015.activeMasterManager] hbase.MetaTableAccessor(1692): Updated table hbase:meta state to ENABLED in META 2015-06-17 11:07:43,527 INFO [10.0.0.110:51015.activeMasterManager] master.ServerManager(686): AssignmentManager hasn't finished failover cleanup; waiting 2015-06-17 11:07:43,528 INFO [10.0.0.110:51015.activeMasterManager] master.HMaster(911): hbase:meta with replicaId 0 assigned=1, location=10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,547 INFO [10.0.0.110:51015.activeMasterManager] master.RegionStates(1072): Transition {9ccc7b1e8942a5472c9cf2d844c2fff0 state=OPEN, ts=1434564463542, server=10.0.0.110,50975,1434564429984} to {9ccc7b1e8942a5472c9cf2d844c2fff0 state=CLOSED, ts=1434564463547, server=10.0.0.110,50975,1434564429984} 2015-06-17 11:07:43,548 INFO [10.0.0.110:51015.activeMasterManager] master.AssignmentManager(538): Clean cluster startup. Assigning user regions 2015-06-17 11:07:43,548 INFO [10.0.0.110:51015.activeMasterManager] master.AssignmentManager(680): Assigning 1 region(s) to 10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,549 INFO [10.0.0.110:51015.activeMasterManager] master.RegionStates(1072): Transition {9ccc7b1e8942a5472c9cf2d844c2fff0 state=CLOSED, ts=1434564463547, server=10.0.0.110,50975,1434564429984} to {9ccc7b1e8942a5472c9cf2d844c2fff0 state=PENDING_OPEN, ts=1434564463549, server=10.0.0.110,51015,1434564463167} 2015-06-17 11:07:43,549 INFO [10.0.0.110:51015.activeMasterManager] master.RegionStateStore(207): Updating row hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. with state=PENDING_OPEN&sn=10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,551 INFO [10.0.0.110:51015.activeMasterManager] regionserver.RSRpcServices(1432): Open hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:07:43,552 DEBUG [10.0.0.110:51015.activeMasterManager] master.AssignmentManager(851): Bulk assigning done for 10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,552 DEBUG [RS_OPEN_REGION-10.0.0.110:51015-1] regionserver.HRegion(6070): Opening region: {ENCODED => 9ccc7b1e8942a5472c9cf2d844c2fff0, NAME => 'hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.', STARTKEY => '', ENDKEY => ''} 2015-06-17 11:07:43,552 DEBUG [RS_OPEN_REGION-10.0.0.110:51015-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table namespace 9ccc7b1e8942a5472c9cf2d844c2fff0 2015-06-17 11:07:43,552 DEBUG [RS_OPEN_REGION-10.0.0.110:51015-1] regionserver.HRegion(722): Instantiated hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:07:43,555 INFO [StoreOpener-9ccc7b1e8942a5472c9cf2d844c2fff0-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=4, currentSize=1076648, freeSize=1042885656, maxSize=1043962304, heapSize=1076648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:07:43,555 INFO [StoreOpener-9ccc7b1e8942a5472c9cf2d844c2fff0-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:07:43,561 INFO [10.0.0.110:51015.activeMasterManager] master.AssignmentManager(420): Joined the cluster in 32ms, failover=false 2015-06-17 11:07:43,566 DEBUG [StoreOpener-9ccc7b1e8942a5472c9cf2d844c2fff0-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/info/b88c94ff75ab4fafba9b040819600c6e, isReference=false, isBulkLoadResult=false, seqid=8, majorCompaction=false 2015-06-17 11:07:43,567 DEBUG [10.0.0.110:51015.activeMasterManager] zookeeper.ZKUtil(486): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2015-06-17 11:07:43,568 DEBUG [10.0.0.110:51015.activeMasterManager] hbase.ZKNamespaceManager(196): Updating namespace cache from node default with data: \x0A\x07default 2015-06-17 11:07:43,568 DEBUG [10.0.0.110:51015.activeMasterManager] hbase.ZKNamespaceManager(196): Updating namespace cache from node hbase with data: \x0A\x05hbase 2015-06-17 11:07:43,568 DEBUG [RS_OPEN_REGION-10.0.0.110:51015-1] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0 2015-06-17 11:07:43,570 DEBUG [RS_OPEN_REGION-10.0.0.110:51015-1] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:07:43,574 DEBUG [RS_OPEN_REGION-10.0.0.110:51015-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/recovered.edits/15.seqid to file, newSeqId=15, maxSeqId=14 2015-06-17 11:07:43,575 INFO [RS_OPEN_REGION-10.0.0.110:51015-1] regionserver.HRegion(848): Onlined 9ccc7b1e8942a5472c9cf2d844c2fff0; next sequenceid=15 2015-06-17 11:07:43,576 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] regionserver.HRegionServer(1893): Post open deploy tasks for hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:07:43,576 DEBUG [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.AssignmentManager(2812): Got transition OPENED for {9ccc7b1e8942a5472c9cf2d844c2fff0 state=PENDING_OPEN, ts=1434564463549, server=10.0.0.110,51015,1434564463167} from 10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,576 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStates(1072): Transition {9ccc7b1e8942a5472c9cf2d844c2fff0 state=PENDING_OPEN, ts=1434564463549, server=10.0.0.110,51015,1434564463167} to {9ccc7b1e8942a5472c9cf2d844c2fff0 state=OPEN, ts=1434564463576, server=10.0.0.110,51015,1434564463167} 2015-06-17 11:07:43,577 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStateStore(207): Updating row hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. with state=OPEN&openSeqNum=15&server=10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,578 DEBUG [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStates(417): Onlined 9ccc7b1e8942a5472c9cf2d844c2fff0 on 10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,578 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStates(429): Offlined 9ccc7b1e8942a5472c9cf2d844c2fff0 from 10.0.0.110,50975,1434564429984 2015-06-17 11:07:43,579 DEBUG [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] regionserver.HRegionServer(1920): Finished post open deploy task for hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:07:43,579 DEBUG [RS_OPEN_REGION-10.0.0.110:51015-1] handler.OpenRegionHandler(125): Opened hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. on 10.0.0.110,51015,1434564463167 2015-06-17 11:07:43,647 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:43,647 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51015: callId: 821 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:43,648 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4157 ms ago, cancelled=false, msg= 2015-06-17 11:07:43,696 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=96, retries=350, started=33248 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:43,850 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:43,850 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51015: callId: 823 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:43,850 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4359 ms ago, cancelled=false, msg= 2015-06-17 11:07:43,897 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=97, retries=350, started=33449 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:44,054 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:44,054 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51015: callId: 825 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:44,054 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4563 ms ago, cancelled=false, msg= 2015-06-17 11:07:44,098 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=98, retries=350, started=33650 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:44,257 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:44,257 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51015: callId: 827 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:44,257 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4766 ms ago, cancelled=false, msg= 2015-06-17 11:07:44,298 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=99, retries=350, started=33850 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:44,439 DEBUG [AsyncRpcChannel-pool2-t3] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2015-06-17 11:07:44,440 DEBUG [RpcServer.listener,port=51015] ipc.RpcServer$Listener(840): RpcServer.listener,port=51015: connection from 10.0.0.110:51032; # active connections: 3 2015-06-17 11:07:44,440 INFO [RpcServer.reader=3,bindAddress=10.0.0.110,port=51015] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51032 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:07:44,459 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:44,459 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51015: callId: 829 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:44,460 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4969 ms ago, cancelled=false, msg= 2015-06-17 11:07:44,499 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=100, retries=350, started=34051 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:44,661 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:44,661 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51015: callId: 831 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:44,661 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5170 ms ago, cancelled=false, msg= 2015-06-17 11:07:44,701 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=101, retries=350, started=34253 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:44,864 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:44,864 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51015: callId: 833 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:44,864 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5373 ms ago, cancelled=false, msg= 2015-06-17 11:07:44,865 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564459491, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:07:44,902 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=102, retries=350, started=34454 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:45,067 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:45,067 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51015: callId: 835 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:45,070 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:45,070 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51015: callId: 837 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:45,074 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:45,074 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51015: callId: 839 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:45,075 DEBUG [10.0.0.110,50905,1434564404764_ChoreService_1] compactions.PressureAwareCompactionThroughputController(148): compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-06-17 11:07:45,080 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:45,080 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51015: callId: 841 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:45,091 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:45,091 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51015: callId: 843 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:45,103 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=103, retries=350, started=34655 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:45,112 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:45,113 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51015: callId: 845 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:45,154 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:45,154 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51015: callId: 847 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:45,255 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:45,255 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51015: callId: 849 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:45,303 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=104, retries=350, started=34855 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:45,358 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:45,358 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51015: callId: 851 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:45,460 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:45,460 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51015: callId: 853 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:45,505 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=105, retries=350, started=35057 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:45,562 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:45,562 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51015: callId: 855 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:45,563 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=497 ms ago, cancelled=false, msg= 2015-06-17 11:07:45,705 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=106, retries=350, started=35257 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:45,764 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:45,764 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51015: callId: 857 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:45,764 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=698 ms ago, cancelled=false, msg= 2015-06-17 11:07:45,907 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=107, retries=350, started=35459 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:45,967 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:45,967 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51015: callId: 859 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:45,967 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=901 ms ago, cancelled=false, msg= 2015-06-17 11:07:46,109 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=108, retries=350, started=35661 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:46,168 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:46,169 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51015: callId: 861 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:46,169 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1103 ms ago, cancelled=false, msg= 2015-06-17 11:07:46,309 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=109, retries=350, started=35861 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:46,372 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:46,372 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51015: callId: 863 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:46,373 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1307 ms ago, cancelled=false, msg= 2015-06-17 11:07:46,510 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=110, retries=350, started=36062 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:46,574 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:46,574 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51015: callId: 865 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:46,574 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1508 ms ago, cancelled=false, msg= 2015-06-17 11:07:46,710 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=111, retries=350, started=36262 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:46,776 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:46,776 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51015: callId: 867 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:46,776 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1710 ms ago, cancelled=false, msg= 2015-06-17 11:07:46,912 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=112, retries=350, started=36464 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:46,978 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:46,978 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51015: callId: 869 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:46,979 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1913 ms ago, cancelled=false, msg= 2015-06-17 11:07:47,113 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=113, retries=350, started=36665 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:47,181 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:47,181 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51015: callId: 871 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:47,181 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2115 ms ago, cancelled=false, msg= 2015-06-17 11:07:47,315 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=114, retries=350, started=36867 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:47,385 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:47,385 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51015: callId: 873 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:47,385 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2319 ms ago, cancelled=false, msg= 2015-06-17 11:07:47,516 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=115, retries=350, started=37068 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:47,586 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:47,586 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51015: callId: 875 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:47,587 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2521 ms ago, cancelled=false, msg= 2015-06-17 11:07:47,718 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=116, retries=350, started=37270 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:47,790 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:47,790 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51015: callId: 877 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:47,790 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2724 ms ago, cancelled=false, msg= 2015-06-17 11:07:47,919 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=117, retries=350, started=37471 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:47,992 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:47,992 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51015: callId: 879 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:47,992 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2926 ms ago, cancelled=false, msg= 2015-06-17 11:07:48,119 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=118, retries=350, started=37671 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:48,194 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:48,194 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51015: callId: 881 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:48,195 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3129 ms ago, cancelled=false, msg= 2015-06-17 11:07:48,320 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=119, retries=350, started=37872 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:48,396 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:48,396 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51015: callId: 883 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:48,396 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3330 ms ago, cancelled=false, msg= 2015-06-17 11:07:48,521 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=120, retries=350, started=38073 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:48,598 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:48,598 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51015: callId: 885 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:48,599 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3533 ms ago, cancelled=false, msg= 2015-06-17 11:07:48,723 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=121, retries=350, started=38275 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:48,800 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:48,800 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51015: callId: 887 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:48,801 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3734 ms ago, cancelled=false, msg= 2015-06-17 11:07:48,924 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=122, retries=350, started=38476 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:49,003 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:49,003 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51015: callId: 889 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:49,003 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3937 ms ago, cancelled=false, msg= 2015-06-17 11:07:49,127 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=123, retries=350, started=38679 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:49,204 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:49,204 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51015: callId: 891 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:49,205 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4139 ms ago, cancelled=false, msg= 2015-06-17 11:07:49,327 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=124, retries=350, started=38879 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:49,406 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:49,406 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51015: callId: 893 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:49,407 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4341 ms ago, cancelled=false, msg= 2015-06-17 11:07:49,527 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=125, retries=350, started=39079 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:49,609 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:49,609 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51015: callId: 895 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:49,609 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4543 ms ago, cancelled=false, msg= 2015-06-17 11:07:49,729 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=126, retries=350, started=39281 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:49,810 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:49,811 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51015: callId: 897 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:49,811 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4745 ms ago, cancelled=false, msg= 2015-06-17 11:07:49,929 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=127, retries=350, started=39481 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:50,013 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:50,013 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51015: callId: 899 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:50,014 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4948 ms ago, cancelled=false, msg= 2015-06-17 11:07:50,130 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=128, retries=350, started=39682 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:50,216 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:50,216 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51015: callId: 901 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:50,216 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5150 ms ago, cancelled=false, msg= 2015-06-17 11:07:50,330 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=129, retries=350, started=39882 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:50,419 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:50,419 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51015: callId: 903 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:50,419 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5353 ms ago, cancelled=false, msg= 2015-06-17 11:07:50,420 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564465066, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:07:50,532 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=130, retries=350, started=40084 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:50,623 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:50,623 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51015: callId: 905 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:50,626 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:50,626 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51015: callId: 907 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:50,631 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:50,631 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51015: callId: 909 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:50,637 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:50,637 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51015: callId: 911 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:50,648 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:50,649 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51015: callId: 913 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:50,670 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:50,670 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51015: callId: 915 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:50,712 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:50,712 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51015: callId: 917 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:50,733 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=131, retries=350, started=40285 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:50,814 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:50,814 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51015: callId: 919 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:50,916 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:50,916 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51015: callId: 921 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:50,935 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=132, retries=350, started=40487 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:51,018 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:51,018 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51015: callId: 923 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:51,120 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:51,120 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51015: callId: 925 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:51,120 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=498 ms ago, cancelled=false, msg= 2015-06-17 11:07:51,136 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=133, retries=350, started=40688 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:51,324 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:51,324 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51015: callId: 927 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:51,325 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=703 ms ago, cancelled=false, msg= 2015-06-17 11:07:51,337 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=134, retries=350, started=40889 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:51,528 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:51,528 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51015: callId: 929 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:51,529 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=907 ms ago, cancelled=false, msg= 2015-06-17 11:07:51,538 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=135, retries=350, started=41090 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:51,731 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:51,731 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51015: callId: 931 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:51,731 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1109 ms ago, cancelled=false, msg= 2015-06-17 11:07:51,740 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=136, retries=350, started=41292 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:51,932 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:51,933 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51015: callId: 933 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:51,933 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1311 ms ago, cancelled=false, msg= 2015-06-17 11:07:51,941 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=137, retries=350, started=41493 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:52,134 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:52,134 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51015: callId: 935 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:52,135 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1513 ms ago, cancelled=false, msg= 2015-06-17 11:07:52,141 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=138, retries=350, started=41693 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:52,338 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:52,339 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51015: callId: 937 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:52,339 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1717 ms ago, cancelled=false, msg= 2015-06-17 11:07:52,341 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=139, retries=350, started=41893 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:52,541 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:52,541 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51015: callId: 939 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:52,542 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1920 ms ago, cancelled=false, msg= 2015-06-17 11:07:52,543 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=140, retries=350, started=42095 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:52,744 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=141, retries=350, started=42296 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:52,745 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:52,745 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51015: callId: 941 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:52,745 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2123 ms ago, cancelled=false, msg= 2015-06-17 11:07:52,944 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=142, retries=350, started=42496 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:52,947 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:52,947 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51015: callId: 943 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:52,948 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2326 ms ago, cancelled=false, msg= 2015-06-17 11:07:53,145 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=143, retries=350, started=42697 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:53,150 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:53,150 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51015: callId: 945 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:53,150 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2528 ms ago, cancelled=false, msg= 2015-06-17 11:07:53,210 DEBUG [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(188): Sleeping for 4621 before executing next master restart count=4 2015-06-17 11:07:53,346 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=144, retries=350, started=42898 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:53,352 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:53,352 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51015: callId: 947 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:53,352 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2730 ms ago, cancelled=false, msg= 2015-06-17 11:07:53,549 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=145, retries=350, started=43101 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:53,555 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:53,555 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51015: callId: 949 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:53,555 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2933 ms ago, cancelled=false, msg= 2015-06-17 11:07:53,751 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=146, retries=350, started=43303 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:53,757 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:53,757 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51015: callId: 951 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:53,757 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3135 ms ago, cancelled=false, msg= 2015-06-17 11:07:53,952 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=147, retries=350, started=43504 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:53,959 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:53,960 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51015: callId: 953 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:53,960 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3338 ms ago, cancelled=false, msg= 2015-06-17 11:07:54,152 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=148, retries=350, started=43704 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:54,163 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:54,163 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51015: callId: 955 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:54,164 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3542 ms ago, cancelled=false, msg= 2015-06-17 11:07:54,354 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=149, retries=350, started=43905 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:54,366 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:54,366 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51015: callId: 957 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:54,367 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3745 ms ago, cancelled=false, msg= 2015-06-17 11:07:54,555 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=150, retries=350, started=44107 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:54,568 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:54,568 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51015: callId: 959 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:54,568 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3946 ms ago, cancelled=false, msg= 2015-06-17 11:07:54,756 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=151, retries=350, started=44308 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:54,770 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:54,771 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51015: callId: 961 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:54,771 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4149 ms ago, cancelled=false, msg= 2015-06-17 11:07:54,957 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=152, retries=350, started=44509 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:54,974 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:54,974 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51015: callId: 963 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:54,975 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4353 ms ago, cancelled=false, msg= 2015-06-17 11:07:55,159 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=153, retries=350, started=44711 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:55,177 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:55,177 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51015: callId: 965 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:55,178 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4556 ms ago, cancelled=false, msg= 2015-06-17 11:07:55,361 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=154, retries=350, started=44913 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:55,380 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:55,380 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51015: callId: 967 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:55,381 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4759 ms ago, cancelled=false, msg= 2015-06-17 11:07:55,563 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=155, retries=350, started=45115 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:55,584 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:55,584 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51015: callId: 969 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:55,585 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4963 ms ago, cancelled=false, msg= 2015-06-17 11:07:55,765 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=156, retries=350, started=45317 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:55,788 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:55,788 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51015: callId: 971 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:55,788 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5166 ms ago, cancelled=false, msg= 2015-06-17 11:07:55,966 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=157, retries=350, started=45518 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:55,991 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:55,991 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51015: callId: 973 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:55,992 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5370 ms ago, cancelled=false, msg= 2015-06-17 11:07:55,992 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564470622, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:07:56,167 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=158, retries=350, started=45719 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:56,196 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:56,196 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51015: callId: 975 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:56,199 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:56,200 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51015: callId: 977 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:56,204 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:56,204 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51015: callId: 979 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:56,211 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:56,211 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51015: callId: 981 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:56,222 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:56,222 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51015: callId: 983 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:56,243 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:56,243 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51015: callId: 985 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:56,285 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:56,285 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51015: callId: 987 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:56,369 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=159, retries=350, started=45920 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:56,387 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:56,387 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51015: callId: 989 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:56,489 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:56,490 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51015: callId: 991 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:56,570 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=160, retries=350, started=46122 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:56,591 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:56,591 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51015: callId: 993 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:56,693 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:56,693 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51015: callId: 995 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:56,693 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=498 ms ago, cancelled=false, msg= 2015-06-17 11:07:56,770 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=161, retries=350, started=46322 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:56,895 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:56,895 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51015: callId: 997 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:56,895 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=700 ms ago, cancelled=false, msg= 2015-06-17 11:07:56,970 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=162, retries=350, started=46522 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:57,096 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:57,096 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51015: callId: 999 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:57,097 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=902 ms ago, cancelled=false, msg= 2015-06-17 11:07:57,171 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=163, retries=350, started=46723 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:57,298 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:57,298 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51015: callId: 1001 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:57,299 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1104 ms ago, cancelled=false, msg= 2015-06-17 11:07:57,373 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=164, retries=350, started=46925 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:57,501 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:57,501 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51015: callId: 1003 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:57,502 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1307 ms ago, cancelled=false, msg= 2015-06-17 11:07:57,574 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=165, retries=350, started=47126 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:57,704 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:57,704 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51015: callId: 1005 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:57,705 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1510 ms ago, cancelled=false, msg= 2015-06-17 11:07:57,774 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=166, retries=350, started=47326 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:57,833 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(197): executing master kill 2015-06-17 11:07:57,833 INFO [MasterRestartThread] hbase.MiniHBaseCluster(422): Aborting 10.0.0.110,51015,1434564463167 2015-06-17 11:07:57,833 FATAL [MasterRestartThread] master.HMaster(1953): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2015-06-17 11:07:57,833 FATAL [MasterRestartThread] master.HMaster(1956): Aborting for tests java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:423) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:270) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$5.run(TestMasterRestartWithProcedures.java:198) 2015-06-17 11:07:57,834 INFO [MasterRestartThread] regionserver.HRegionServer(1859): STOPPED: Aborting for tests 2015-06-17 11:07:57,834 INFO [MasterRestartThread] hbase.LocalHBaseCluster(379): Waiting on 10.0.0.110,51015,1434564463167 2015-06-17 11:07:57,834 INFO [M:0;10.0.0.110:51015] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2015-06-17 11:07:57,834 INFO [M:0;10.0.0.110:51015] regionserver.HeapMemoryManager(201): Stoping HeapMemoryTuner chore. 2015-06-17 11:07:57,834 INFO [SplitLogWorker-10.0.0.110:51015] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2015-06-17 11:07:57,834 INFO [M:0;10.0.0.110:51015] procedure2.ProcedureExecutor(490): Stopping the procedure executor 2015-06-17 11:07:57,834 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2015-06-17 11:07:57,834 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2015-06-17 11:07:57,834 INFO [SplitLogWorker-10.0.0.110:51015] regionserver.SplitLogWorker(155): SplitLogWorker 10.0.0.110,51015,1434564463167 exiting 2015-06-17 11:07:57,834 INFO [M:0;10.0.0.110:51015] wal.WALProcedureStore(159): Stopping the WAL Procedure Store 2015-06-17 11:07:57,841 INFO [IPC Server handler 3 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741868_1044{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 0 2015-06-17 11:07:57,842 INFO [M:0;10.0.0.110:51015] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2015-06-17 11:07:57,842 INFO [M:0;10.0.0.110:51015] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2015-06-17 11:07:57,842 INFO [M:0;10.0.0.110:51015] regionserver.HRegionServer(1021): stopping server 10.0.0.110,51015,1434564463167 2015-06-17 11:07:57,842 DEBUG [M:0;10.0.0.110:51015] zookeeper.MetaTableLocator(617): Stopping MetaTableLocator 2015-06-17 11:07:57,842 DEBUG [RS_CLOSE_REGION-10.0.0.110:51015-0] handler.CloseRegionHandler(90): Processing close of hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:07:57,842 DEBUG [RS_CLOSE_REGION-10.0.0.110:51015-0] regionserver.HRegion(1365): Closing hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.: disabling compactions & flushes 2015-06-17 11:07:57,842 DEBUG [RS_CLOSE_REGION-10.0.0.110:51015-0] regionserver.HRegion(1392): Updates disabled for region hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:07:57,843 INFO [M:0;10.0.0.110:51015] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320019 2015-06-17 11:07:57,843 INFO [StoreCloserThread-hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.-1] regionserver.HStore(870): Closed info 2015-06-17 11:07:57,844 DEBUG [M:0;10.0.0.110:51015] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:07:57,844 INFO [M:0;10.0.0.110:51015] regionserver.CompactSplitThread(394): Waiting for Split Thread to finish... 2015-06-17 11:07:57,844 INFO [M:0;10.0.0.110:51015] regionserver.CompactSplitThread(394): Waiting for Merge Thread to finish... 2015-06-17 11:07:57,844 DEBUG [AsyncRpcChannel-pool2-t1] ipc.AsyncRpcChannel$8(574): IPC Client (-603036063) connection to /10.0.0.110:51015 from ndimiduk.hfs.4: closed 2015-06-17 11:07:57,844 DEBUG [RpcServer.reader=2,bindAddress=10.0.0.110,port=51015] ipc.RpcServer$Listener(872): RpcServer.listener,port=51015: DISCONNECTING client 10.0.0.110:51029 because read count=-1. Number of active connections: 3 2015-06-17 11:07:57,844 INFO [M:0;10.0.0.110:51015] regionserver.CompactSplitThread(394): Waiting for Large Compaction Thread to finish... 2015-06-17 11:07:57,844 DEBUG [AsyncRpcChannel-pool2-t2] ipc.AsyncRpcChannel$8(574): IPC Client (1441817670) connection to /10.0.0.110:50975 from ndimiduk.hfs.4: closed 2015-06-17 11:07:57,845 INFO [M:0;10.0.0.110:51015] regionserver.CompactSplitThread(394): Waiting for Small Compaction Thread to finish... 2015-06-17 11:07:57,845 INFO [M:0;10.0.0.110:51015] regionserver.HRegionServer(1243): Waiting on 2 regions to close 2015-06-17 11:07:57,845 DEBUG [M:0;10.0.0.110:51015] regionserver.HRegionServer(1247): {9ccc7b1e8942a5472c9cf2d844c2fff0=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., 1588230740=hbase:meta,,1.1588230740} 2015-06-17 11:07:57,845 DEBUG [RS_CLOSE_META-10.0.0.110:51015-0] handler.CloseRegionHandler(90): Processing close of hbase:meta,,1.1588230740 2015-06-17 11:07:57,845 DEBUG [RS_CLOSE_META-10.0.0.110:51015-0] regionserver.HRegion(1365): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2015-06-17 11:07:57,845 DEBUG [RS_CLOSE_META-10.0.0.110:51015-0] regionserver.HRegion(1392): Updates disabled for region hbase:meta,,1.1588230740 2015-06-17 11:07:57,846 INFO [RS_CLOSE_META-10.0.0.110:51015-0] regionserver.HRegion(2111): Flushing 2/2 column families, memstore=1.75 KB 2015-06-17 11:07:57,850 DEBUG [RS_CLOSE_REGION-10.0.0.110:51015-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/recovered.edits/17.seqid to file, newSeqId=17, maxSeqId=15 2015-06-17 11:07:57,850 INFO [RS_CLOSE_REGION-10.0.0.110:51015-0] regionserver.HRegion(1486): Closed hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:07:57,851 DEBUG [RS_CLOSE_REGION-10.0.0.110:51015-0] handler.CloseRegionHandler(122): Closed hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:07:57,854 INFO [IPC Server handler 1 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741869_1045{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 0 2015-06-17 11:07:57,854 INFO [RS_CLOSE_META-10.0.0.110:51015-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=80000131, memsize=1.4 K, hasBloomFilter=false, into tmp file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/96eb1546e8d44dd081fd3c2be70b7d3e 2015-06-17 11:07:57,865 INFO [IPC Server handler 2 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741870_1046{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 4754 2015-06-17 11:07:57,907 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:57,907 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51015: callId: 1007 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:57,908 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1713 ms ago, cancelled=false, msg= 2015-06-17 11:07:57,976 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=167, retries=350, started=47528 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:58,112 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:58,112 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51015: callId: 1009 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:58,113 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1918 ms ago, cancelled=false, msg= 2015-06-17 11:07:58,177 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=168, retries=350, started=47729 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:58,204 INFO [10.0.0.110,51015,1434564463167_splitLogManager__ChoreService_1] hbase.ScheduledChore(179): Chore: SplitLogManager Timeout Monitor was stopped 2015-06-17 11:07:58,261 INFO [10.0.0.110,51015,1434564463167_ChoreService_1] hbase.ScheduledChore(179): Chore: 10.0.0.110,51015,1434564463167-MemstoreFlusherChore was stopped 2015-06-17 11:07:58,264 INFO [master//10.0.0.110:0.logRoller] regionserver.LogRoller(159): LogRoller exiting. 2015-06-17 11:07:58,266 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(146): master//10.0.0.110:0.leaseChecker closing leases 2015-06-17 11:07:58,266 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(149): master//10.0.0.110:0.leaseChecker closed leases 2015-06-17 11:07:58,267 INFO [RS_CLOSE_META-10.0.0.110:51015-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=80000131, memsize=352, hasBloomFilter=false, into tmp file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/c394aa982f5044639d5cdde07e27d56e 2015-06-17 11:07:58,276 DEBUG [RS_CLOSE_META-10.0.0.110:51015-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/96eb1546e8d44dd081fd3c2be70b7d3e as hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/96eb1546e8d44dd081fd3c2be70b7d3e 2015-06-17 11:07:58,283 INFO [RS_CLOSE_META-10.0.0.110:51015-0] regionserver.HStore(963): Added hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/96eb1546e8d44dd081fd3c2be70b7d3e, entries=6, sequenceid=80000131, filesize=5.3 K 2015-06-17 11:07:58,284 DEBUG [RS_CLOSE_META-10.0.0.110:51015-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/c394aa982f5044639d5cdde07e27d56e as hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/c394aa982f5044639d5cdde07e27d56e 2015-06-17 11:07:58,291 INFO [RS_CLOSE_META-10.0.0.110:51015-0] regionserver.HStore(963): Added hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/c394aa982f5044639d5cdde07e27d56e, entries=2, sequenceid=80000131, filesize=4.6 K 2015-06-17 11:07:58,292 INFO [RS_CLOSE_META-10.0.0.110:51015-0] regionserver.HRegion(2400): Finished memstore flush of ~1.75 KB/1792, currentsize=0 B/0 for region hbase:meta,,1.1588230740 in 446ms, sequenceid=80000131, compaction requested=true 2015-06-17 11:07:58,293 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(870): Closed info 2015-06-17 11:07:58,295 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(870): Closed table 2015-06-17 11:07:58,301 DEBUG [RS_CLOSE_META-10.0.0.110:51015-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/recovered.edits/80000134.seqid to file, newSeqId=80000134, maxSeqId=80000120 2015-06-17 11:07:58,301 DEBUG [RS_CLOSE_META-10.0.0.110:51015-0] coprocessor.CoprocessorHost(269): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2015-06-17 11:07:58,302 INFO [RS_CLOSE_META-10.0.0.110:51015-0] regionserver.HRegion(1486): Closed hbase:meta,,1.1588230740 2015-06-17 11:07:58,302 DEBUG [RS_CLOSE_META-10.0.0.110:51015-0] handler.CloseRegionHandler(122): Closed hbase:meta,,1.1588230740 2015-06-17 11:07:58,311 INFO [RS_OPEN_META-10.0.0.110:51015-0-MetaLogRoller] regionserver.LogRoller(159): LogRoller exiting. 2015-06-17 11:07:58,318 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:58,319 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51015] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51015: callId: 1011 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51022 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:58,320 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2125 ms ago, cancelled=false, msg= 2015-06-17 11:07:58,380 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=169, retries=350, started=47932 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:58,449 INFO [M:0;10.0.0.110:51015] regionserver.HRegionServer(1049): stopping server 10.0.0.110,51015,1434564463167; all regions closed. 2015-06-17 11:07:58,449 DEBUG [M:0;10.0.0.110:51015] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51015,1434564463167 2015-06-17 11:07:58,452 INFO [IPC Server handler 2 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741867_1043{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 83 2015-06-17 11:07:58,456 DEBUG [M:0;10.0.0.110:51015] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:07:58,456 INFO [M:0;10.0.0.110:51015] wal.FSHLog(977): Closed WAL: FSHLog 10.0.0.110%2C51015%2C1434564463167..meta:.meta(num 1434564463306) 2015-06-17 11:07:58,456 DEBUG [M:0;10.0.0.110:51015] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51015,1434564463167 2015-06-17 11:07:58,458 INFO [IPC Server handler 8 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741866_1042{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 83 2015-06-17 11:07:58,460 DEBUG [M:0;10.0.0.110:51015] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:07:58,461 INFO [M:0;10.0.0.110:51015] wal.FSHLog(977): Closed WAL: FSHLog 10.0.0.110%2C51015%2C1434564463167.default:(num 1434564463252) 2015-06-17 11:07:58,461 DEBUG [M:0;10.0.0.110:51015] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:07:58,461 INFO [M:0;10.0.0.110:51015] regionserver.Leases(146): M:0;10.0.0.110:51015 closing leases 2015-06-17 11:07:58,461 INFO [M:0;10.0.0.110:51015] regionserver.Leases(149): M:0;10.0.0.110:51015 closed leases 2015-06-17 11:07:58,461 INFO [M:0;10.0.0.110:51015] hbase.ChoreService(302): Chore service for: 10.0.0.110,51015,1434564463167 had [[ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.0.0.110,51015,1434564463167-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.0.0.110,51015,1434564463167-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-10.0.0.110:51015 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region 10.0.0.110,51015,1434564463167 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS]] on shutdown 2015-06-17 11:07:58,463 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/10.0.0.110,51015,1434564463167 2015-06-17 11:07:58,463 INFO [M:0;10.0.0.110:51015] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a232001c 2015-06-17 11:07:58,464 DEBUG [M:0;10.0.0.110:51015] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:07:58,464 DEBUG [M:0;10.0.0.110:51015] master.HMaster(1074): Stopping service threads 2015-06-17 11:07:58,465 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:07:58,465 INFO [M:0;10.0.0.110:51015] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a232001a 2015-06-17 11:07:58,465 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:07:58,466 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(488): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:07:58,466 DEBUG [M:0;10.0.0.110:51015] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:07:58,466 DEBUG [main-EventThread] zookeeper.ZKUtil(488): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:07:58,466 INFO [M:0;10.0.0.110:51015] hbase.ChoreService(302): Chore service for: 10.0.0.110,51015,1434564463167_splitLogManager_ had [] on shutdown 2015-06-17 11:07:58,466 INFO [M:0;10.0.0.110:51015] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2015-06-17 11:07:58,466 INFO [M:0;10.0.0.110:51015] ipc.RpcServer(2219): Stopping server on 51015 2015-06-17 11:07:58,467 INFO [RpcServer.listener,port=51015] ipc.RpcServer$Listener(776): RpcServer.listener,port=51015: stopping 2015-06-17 11:07:58,468 INFO [RpcServer.responder] ipc.RpcServer$Responder(1018): RpcServer.responder: stopped 2015-06-17 11:07:58,468 INFO [RpcServer.responder] ipc.RpcServer$Responder(921): RpcServer.responder: stopping 2015-06-17 11:07:58,469 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,51015,1434564463167 2015-06-17 11:07:58,469 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,51015,1434564463167 2015-06-17 11:07:58,469 INFO [MasterRestartThread-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [10.0.0.110,51015,1434564463167] 2015-06-17 11:07:58,469 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51015-0x14e02b3a2320018, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:07:58,470 INFO [M:0;10.0.0.110:51015] regionserver.HRegionServer(1093): stopping server 10.0.0.110,51015,1434564463167; zookeeper connection closed. 2015-06-17 11:07:58,470 INFO [M:0;10.0.0.110:51015] regionserver.HRegionServer(1096): M:0;10.0.0.110:51015 exiting 2015-06-17 11:07:58,470 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(200): executing master restart 2015-06-17 11:07:58,470 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:58,471 INFO [main-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(121): /hbase/rs/10.0.0.110,51015,1434564463167 znode expired, triggering replicatorRemoved event 2015-06-17 11:07:58,471 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:07:58,471 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:58,489 INFO [MasterRestartThread] client.ConnectionUtils(105): master//10.0.0.110:0 server-side HConnection retries=350 2015-06-17 11:07:58,489 INFO [MasterRestartThread] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2015-06-17 11:07:58,491 INFO [MasterRestartThread] ipc.RpcServer$Listener(598): master//10.0.0.110:0: started 10 reader(s). 2015-06-17 11:07:58,493 INFO [MasterRestartThread] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:07:58,494 INFO [MasterRestartThread] zookeeper.RecoverableZooKeeper(121): Process identifier=master:51047 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:07:58,496 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:510470x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:07:58,497 DEBUG [MasterRestartThread] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-17 11:07:58,498 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(578): master:51047-0x14e02b3a232001d connected 2015-06-17 11:07:58,499 DEBUG [MasterRestartThread] zookeeper.ZKUtil(488): master:51047-0x14e02b3a232001d, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:07:58,500 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:51047-0x14e02b3a232001d, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2015-06-17 11:07:58,500 INFO [RpcServer.responder] ipc.RpcServer$Responder(917): RpcServer.responder: starting 2015-06-17 11:07:58,500 INFO [RpcServer.listener,port=51047] ipc.RpcServer$Listener(729): RpcServer.listener,port=51047: starting 2015-06-17 11:07:58,500 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=0 queue=0 2015-06-17 11:07:58,500 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=1 queue=0 2015-06-17 11:07:58,501 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=2 queue=0 2015-06-17 11:07:58,501 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=3 queue=0 2015-06-17 11:07:58,501 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=4 queue=0 2015-06-17 11:07:58,501 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=0 queue=0 2015-06-17 11:07:58,501 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=1 queue=1 2015-06-17 11:07:58,501 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=2 queue=0 2015-06-17 11:07:58,501 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=3 queue=1 2015-06-17 11:07:58,501 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=4 queue=0 2015-06-17 11:07:58,502 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=5 queue=1 2015-06-17 11:07:58,502 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=6 queue=0 2015-06-17 11:07:58,502 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=7 queue=1 2015-06-17 11:07:58,502 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=8 queue=0 2015-06-17 11:07:58,502 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=9 queue=1 2015-06-17 11:07:58,502 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=10 queue=0 2015-06-17 11:07:58,502 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=11 queue=1 2015-06-17 11:07:58,502 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=12 queue=0 2015-06-17 11:07:58,502 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=13 queue=1 2015-06-17 11:07:58,503 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=14 queue=0 2015-06-17 11:07:58,503 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=15 queue=1 2015-06-17 11:07:58,503 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=16 queue=0 2015-06-17 11:07:58,503 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=17 queue=1 2015-06-17 11:07:58,503 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=18 queue=0 2015-06-17 11:07:58,503 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=19 queue=1 2015-06-17 11:07:58,503 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=0 queue=0 2015-06-17 11:07:58,503 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=1 queue=0 2015-06-17 11:07:58,504 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=2 queue=0 2015-06-17 11:07:58,504 INFO [MasterRestartThread] master.HMaster(349): hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034, hbase.cluster.distributed=false 2015-06-17 11:07:58,504 INFO [MasterRestartThread] master.HMaster(1512): Adding backup master ZNode /hbase/backup-masters/10.0.0.110,51047,1434564478492 2015-06-17 11:07:58,505 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:51047-0x14e02b3a232001d, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/10.0.0.110,51047,1434564478492 2015-06-17 11:07:58,506 INFO [M:0;10.0.0.110:51047] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x2a3a5feb connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:07:58,506 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51047-0x14e02b3a232001d, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:07:58,506 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:07:58,507 DEBUG [10.0.0.110:51047.activeMasterManager] zookeeper.ZKUtil(486): master:51047-0x14e02b3a232001d, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:07:58,507 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(486): master:51047-0x14e02b3a232001d, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:07:58,507 INFO [10.0.0.110:51047.activeMasterManager] master.ActiveMasterManager(170): Deleting ZNode for /hbase/backup-masters/10.0.0.110,51047,1434564478492 from backup master directory 2015-06-17 11:07:58,507 DEBUG [MasterRestartThread-EventThread] master.ActiveMasterManager(126): A master is now available 2015-06-17 11:07:58,508 DEBUG [M:0;10.0.0.110:51047-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x2a3a5feb0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:07:58,508 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51047-0x14e02b3a232001d, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/10.0.0.110,51047,1434564478492 2015-06-17 11:07:58,508 WARN [10.0.0.110:51047.activeMasterManager] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:07:58,508 INFO [10.0.0.110:51047.activeMasterManager] master.ActiveMasterManager(179): Registered Active Master=10.0.0.110,51047,1434564478492 2015-06-17 11:07:58,509 DEBUG [M:0;10.0.0.110:51047] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a369d34, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:07:58,509 DEBUG [M:0;10.0.0.110:51047] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:07:58,509 DEBUG [M:0;10.0.0.110:51047] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:07:58,509 DEBUG [M:0;10.0.0.110:51047-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x2a3a5feb-0x14e02b3a232001e connected 2015-06-17 11:07:58,510 INFO [M:0;10.0.0.110:51047] regionserver.HRegionServer(774): ClusterId : b7c4b21b-aab2-44b4-8562-2b61af22ad6c 2015-06-17 11:07:58,510 DEBUG [M:0;10.0.0.110:51047] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2015-06-17 11:07:58,511 DEBUG [M:0;10.0.0.110:51047] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:07:58,511 DEBUG [M:0;10.0.0.110:51047] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2015-06-17 11:07:58,511 DEBUG [M:0;10.0.0.110:51047] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2015-06-17 11:07:58,512 DEBUG [M:0;10.0.0.110:51047] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:07:58,512 DEBUG [M:0;10.0.0.110:51047] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2015-06-17 11:07:58,513 INFO [M:0;10.0.0.110:51047] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2015-06-17 11:07:58,513 INFO [M:0;10.0.0.110:51047] compactions.PressureAwareCompactionThroughputController(175): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2015-06-17 11:07:58,513 INFO [M:0;10.0.0.110:51047] regionserver.HRegionServer$CompactionChecker(1504): CompactionChecker runs every 1sec 2015-06-17 11:07:58,513 DEBUG [M:0;10.0.0.110:51047] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55723b04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=10.0.0.110/10.0.0.110:0 2015-06-17 11:07:58,514 DEBUG [M:0;10.0.0.110:51047] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:07:58,514 DEBUG [M:0;10.0.0.110:51047] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:07:58,514 DEBUG [M:0;10.0.0.110:51047] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:M:0;10.0.0.110:51047 2015-06-17 11:07:58,514 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:07:58,515 DEBUG [M:0;10.0.0.110:51047] zookeeper.ZKUtil(486): master:51047-0x14e02b3a232001d, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51047,1434564478492 2015-06-17 11:07:58,515 INFO [M:0;10.0.0.110:51047] regionserver.RegionServerCoprocessorHost(65): System coprocessor loading is enabled 2015-06-17 11:07:58,515 INFO [M:0;10.0.0.110:51047] regionserver.RegionServerCoprocessorHost(66): Table coprocessor loading is enabled 2015-06-17 11:07:58,515 INFO [M:0;10.0.0.110:51047] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,51047,1434564478492 with port=51047, startcode=1434564478492 2015-06-17 11:07:58,515 DEBUG [M:0;10.0.0.110:51047] regionserver.HRegionServer(2295): Master is not running yet 2015-06-17 11:07:58,515 WARN [M:0;10.0.0.110:51047] regionserver.HRegionServer(900): reportForDuty failed; sleeping and then retrying. 2015-06-17 11:07:58,515 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:58,515 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51047,1434564478492 2015-06-17 11:07:58,520 DEBUG [10.0.0.110:51047.activeMasterManager] util.FSTableDescriptors(807): Current tableInfoPath = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2015-06-17 11:07:58,522 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:07:58,523 DEBUG [10.0.0.110:51047.activeMasterManager] util.FSTableDescriptors(811): TableInfo already exists.. Skipping creation 2015-06-17 11:07:58,524 DEBUG [AsyncRpcChannel-pool2-t4] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service MasterService, sasl=false 2015-06-17 11:07:58,524 DEBUG [RpcServer.listener,port=51047] ipc.RpcServer$Listener(840): RpcServer.listener,port=51047: connection from 10.0.0.110:51051; # active connections: 1 2015-06-17 11:07:58,527 INFO [RpcServer.reader=1,bindAddress=10.0.0.110,port=51047] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51051 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:07:58,528 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51047: callId: 1013 service: MasterService methodName: IsMasterRunning size: 24 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:58,528 INFO [10.0.0.110:51047.activeMasterManager] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:07:58,528 DEBUG [10.0.0.110:51047.activeMasterManager] coordination.ZKSplitLogManagerCoordination(885): Distributed log replay=true 2015-06-17 11:07:58,528 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2333 ms ago, cancelled=false, msg= 2015-06-17 11:07:58,529 INFO [10.0.0.110:51047.activeMasterManager] coordination.ZKSplitLogManagerCoordination(599): Found 0 orphan tasks and 0 rescan nodes 2015-06-17 11:07:58,530 DEBUG [10.0.0.110:51047.activeMasterManager] util.FSTableDescriptors(222): Fetching table descriptors from the filesystem. 2015-06-17 11:07:58,530 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=350, started=14962 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:07:58,541 INFO [10.0.0.110:51047.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x49866344 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:07:58,543 DEBUG [10.0.0.110:51047.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x498663440x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:07:58,543 DEBUG [10.0.0.110:51047.activeMasterManager] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f421265, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:07:58,543 DEBUG [10.0.0.110:51047.activeMasterManager] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:07:58,543 DEBUG [10.0.0.110:51047.activeMasterManager] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:07:58,544 DEBUG [10.0.0.110:51047.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x49866344-0x14e02b3a232001f connected 2015-06-17 11:07:58,544 INFO [10.0.0.110:51047.activeMasterManager] balancer.StochasticLoadBalancer(135): loading config 2015-06-17 11:07:58,545 DEBUG [10.0.0.110:51047.activeMasterManager] zookeeper.ZKUtil(488): master:51047-0x14e02b3a232001d, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2015-06-17 11:07:58,545 DEBUG [10.0.0.110:51047.activeMasterManager] zookeeper.ZKUtil(486): master:51047-0x14e02b3a232001d, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:58,545 DEBUG [10.0.0.110:51047.activeMasterManager] zookeeper.ZKUtil(486): master:51047-0x14e02b3a232001d, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51047,1434564478492 2015-06-17 11:07:58,546 DEBUG [10.0.0.110:51047.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:58,546 DEBUG [10.0.0.110:51047.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,51047,1434564478492 2015-06-17 11:07:58,546 INFO [10.0.0.110:51047.activeMasterManager] master.HMaster(568): Server active/primary master=10.0.0.110,51047,1434564478492, sessionid=0x14e02b3a232001d, setting cluster-up flag (Was=true) 2015-06-17 11:07:58,547 DEBUG [10.0.0.110:51047.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:07:58,548 INFO [10.0.0.110:51047.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2015-06-17 11:07:58,548 DEBUG [10.0.0.110:51047.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,51047,1434564478492 2015-06-17 11:07:58,549 DEBUG [10.0.0.110:51047.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:07:58,550 INFO [10.0.0.110:51047.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2015-06-17 11:07:58,550 DEBUG [10.0.0.110:51047.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,51047,1434564478492 2015-06-17 11:07:58,550 INFO [10.0.0.110:51047.activeMasterManager] master.MasterCoprocessorHost(86): System coprocessor loading is enabled 2015-06-17 11:07:58,550 DEBUG [10.0.0.110:51047.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_OPEN_REGION-10.0.0.110:51047, corePoolSize=5, maxPoolSize=5 2015-06-17 11:07:58,551 DEBUG [10.0.0.110:51047.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_CLOSE_REGION-10.0.0.110:51047, corePoolSize=5, maxPoolSize=5 2015-06-17 11:07:58,551 DEBUG [10.0.0.110:51047.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_SERVER_OPERATIONS-10.0.0.110:51047, corePoolSize=5, maxPoolSize=5 2015-06-17 11:07:58,551 DEBUG [10.0.0.110:51047.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_META_SERVER_OPERATIONS-10.0.0.110:51047, corePoolSize=5, maxPoolSize=5 2015-06-17 11:07:58,551 DEBUG [10.0.0.110:51047.activeMasterManager] executor.ExecutorService(91): Starting executor service name=M_LOG_REPLAY_OPS-10.0.0.110:51047, corePoolSize=10, maxPoolSize=10 2015-06-17 11:07:58,551 DEBUG [10.0.0.110:51047.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_TABLE_OPERATIONS-10.0.0.110:51047, corePoolSize=1, maxPoolSize=1 2015-06-17 11:07:58,551 INFO [10.0.0.110:51047.activeMasterManager] procedure2.ProcedureExecutor(446): Starting procedure executor threads=9 2015-06-17 11:07:58,551 INFO [10.0.0.110:51047.activeMasterManager] wal.WALProcedureStore(214): Starting WAL Procedure Store lease recovery 2015-06-17 11:07:58,552 INFO [10.0.0.110:51047.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log 2015-06-17 11:07:58,553 INFO [10.0.0.110:51047.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log after 1ms 2015-06-17 11:07:58,553 DEBUG [10.0.0.110:51047.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log; isDirectory=false; length=14915; replication=1; blocksize=134217728; modification_time=1434564413352; access_time=1434564405122; owner=ndimiduk; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:07:58,555 INFO [10.0.0.110:51047.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000005.log 2015-06-17 11:07:58,556 INFO [10.0.0.110:51047.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000005.log after 1ms 2015-06-17 11:07:58,556 DEBUG [10.0.0.110:51047.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000005.log; isDirectory=false; length=60; replication=1; blocksize=134217728; modification_time=1434564477841; access_time=1434564463233; owner=ndimiduk.hfs.4; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:07:58,559 DEBUG [10.0.0.110:51047.activeMasterManager] wal.WALProcedureStore(610): Roll new state log: 6 2015-06-17 11:07:58,560 INFO [10.0.0.110:51047.activeMasterManager] wal.WALProcedureStore(237): Lease acquired for flushLogId: 6 2015-06-17 11:07:58,561 INFO [10.0.0.110:51047.activeMasterManager] wal.ProcedureWALFormatReader(150): No active entry found in state log hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000005.log. removing it 2015-06-17 11:07:58,562 DEBUG [10.0.0.110:51047.activeMasterManager] procedure2.ProcedureExecutor$1(278): load procedures maxProcId=7 2015-06-17 11:07:58,563 DEBUG [10.0.0.110:51047.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=RUNNABLE isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=7 state=RUNNABLE 2015-06-17 11:07:58,563 DEBUG [10.0.0.110:51047.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DeleteTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=6 state=FINISHED 2015-06-17 11:07:58,563 DEBUG [10.0.0.110:51047.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:58,564 DEBUG [10.0.0.110:51047.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=5 state=FINISHED 2015-06-17 11:07:58,564 DEBUG [10.0.0.110:51047.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:58,564 DEBUG [10.0.0.110:51047.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: EnableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=4 state=FINISHED 2015-06-17 11:07:58,564 DEBUG [10.0.0.110:51047.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:58,564 DEBUG [10.0.0.110:51047.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=3 state=FINISHED 2015-06-17 11:07:58,564 DEBUG [10.0.0.110:51047.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:58,564 DEBUG [10.0.0.110:51047.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=2 state=FINISHED 2015-06-17 11:07:58,564 DEBUG [10.0.0.110:51047.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:58,565 DEBUG [10.0.0.110:51047.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=hbase:namespace) user=ndimiduk (auth:SIMPLE) id=1 state=FINISHED 2015-06-17 11:07:58,565 DEBUG [10.0.0.110:51047.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:07:58,565 DEBUG [10.0.0.110:51047.activeMasterManager] wal.WALProcedureStore(652): Remove log: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000005.log 2015-06-17 11:07:58,566 INFO [IPC Server handler 1 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741868_1044 127.0.0.1:50896 2015-06-17 11:07:58,567 DEBUG [10.0.0.110:51047.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2015-06-17 11:07:58,568 INFO [10.0.0.110:51047.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:07:58,569 DEBUG [10.0.0.110:51047.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:07:58,570 DEBUG [10.0.0.110:51047.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/replication/rs already exists 2015-06-17 11:07:58,570 DEBUG [10.0.0.110:51047.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2015-06-17 11:07:58,570 DEBUG [10.0.0.110:51047.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner 2015-06-17 11:07:58,571 DEBUG [10.0.0.110:51047.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): replicationLogCleaner-0x14e02b3a2320020 connected 2015-06-17 11:07:58,571 DEBUG [10.0.0.110:51047.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2015-06-17 11:07:58,572 DEBUG [10.0.0.110:51047.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2015-06-17 11:07:58,572 DEBUG [10.0.0.110:51047.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2015-06-17 11:07:58,572 INFO [10.0.0.110:51047.activeMasterManager] master.ServerManager(1011): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 1, timeout of 4500 ms, interval of 1500 ms. 2015-06-17 11:07:58,572 INFO [M:0;10.0.0.110:51047] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,51047,1434564478492 with port=51047, startcode=1434564478492 2015-06-17 11:07:58,572 INFO [M:0;10.0.0.110:51047] master.ServerManager(459): Registering server=10.0.0.110,51047,1434564478492 2015-06-17 11:07:58,572 INFO [M:0;10.0.0.110:51047] regionserver.HRegionServer(1341): Config from master: hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034 2015-06-17 11:07:58,573 INFO [M:0;10.0.0.110:51047] regionserver.HRegionServer(1341): Config from master: fs.defaultFS=hdfs://localhost:50895 2015-06-17 11:07:58,573 INFO [M:0;10.0.0.110:51047] regionserver.HRegionServer(1341): Config from master: hbase.master.info.port=-1 2015-06-17 11:07:58,573 WARN [M:0;10.0.0.110:51047] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:07:58,573 INFO [M:0;10.0.0.110:51047] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:07:58,573 DEBUG [M:0;10.0.0.110:51047] regionserver.HRegionServer(1603): logdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51047,1434564478492 2015-06-17 11:07:58,575 DEBUG [M:0;10.0.0.110:51047] regionserver.Replication(142): ReplicationStatisticsThread 300 2015-06-17 11:07:58,576 INFO [M:0;10.0.0.110:51047] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:07:58,578 INFO [M:0;10.0.0.110:51047] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.0.0.110%2C51047%2C1434564478492.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51047,1434564478492, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:07:58,580 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=170, retries=350, started=48132 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:58,583 INFO [M:0;10.0.0.110:51047] wal.FSHLog(1365): Slow sync cost: 3 ms, current pipeline: [] 2015-06-17 11:07:58,583 INFO [M:0;10.0.0.110:51047] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51047,1434564478492/10.0.0.110%2C51047%2C1434564478492.default.1434564478578 2015-06-17 11:07:58,584 INFO [M:0;10.0.0.110:51047] regionserver.MetricsRegionServerWrapperImpl(113): Computing regionserver metrics every 5000 milliseconds 2015-06-17 11:07:58,584 DEBUG [M:0;10.0.0.110:51047] executor.ExecutorService(91): Starting executor service name=RS_OPEN_REGION-10.0.0.110:51047, corePoolSize=3, maxPoolSize=3 2015-06-17 11:07:58,584 DEBUG [M:0;10.0.0.110:51047] executor.ExecutorService(91): Starting executor service name=RS_OPEN_META-10.0.0.110:51047, corePoolSize=1, maxPoolSize=1 2015-06-17 11:07:58,584 DEBUG [M:0;10.0.0.110:51047] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_REGION-10.0.0.110:51047, corePoolSize=3, maxPoolSize=3 2015-06-17 11:07:58,585 DEBUG [M:0;10.0.0.110:51047] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_META-10.0.0.110:51047, corePoolSize=1, maxPoolSize=1 2015-06-17 11:07:58,585 DEBUG [M:0;10.0.0.110:51047] executor.ExecutorService(91): Starting executor service name=RS_LOG_REPLAY_OPS-10.0.0.110:51047, corePoolSize=2, maxPoolSize=2 2015-06-17 11:07:58,585 DEBUG [M:0;10.0.0.110:51047] executor.ExecutorService(91): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-10.0.0.110:51047, corePoolSize=3, maxPoolSize=3 2015-06-17 11:07:58,586 DEBUG [M:0;10.0.0.110:51047] zookeeper.ZKUtil(486): master:51047-0x14e02b3a232001d, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:07:58,586 DEBUG [M:0;10.0.0.110:51047] zookeeper.ZKUtil(486): master:51047-0x14e02b3a232001d, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51047,1434564478492 2015-06-17 11:07:58,586 INFO [M:0;10.0.0.110:51047] regionserver.ReplicationSourceManager(229): Current list of replicators: [10.0.0.110,50905,1434564404764, 10.0.0.110,51047,1434564478492] other RSs: [10.0.0.110,50905,1434564404764, 10.0.0.110,51047,1434564478492] 2015-06-17 11:07:58,599 INFO [M:0;10.0.0.110:51047] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x4ebc1581 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:07:58,601 DEBUG [M:0;10.0.0.110:51047-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x4ebc15810x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:07:58,602 DEBUG [M:0;10.0.0.110:51047] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodecWithTags@1952d81, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:07:58,602 DEBUG [M:0;10.0.0.110:51047] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:07:58,602 DEBUG [M:0;10.0.0.110:51047] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:07:58,603 DEBUG [M:0;10.0.0.110:51047-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x4ebc1581-0x14e02b3a2320021 connected 2015-06-17 11:07:58,617 INFO [M:0;10.0.0.110:51047] regionserver.HeapMemoryManager(190): Starting HeapMemoryTuner chore. 2015-06-17 11:07:58,617 INFO [SplitLogWorker-10.0.0.110:51047] regionserver.SplitLogWorker(134): SplitLogWorker 10.0.0.110,51047,1434564478492 starting 2015-06-17 11:07:58,617 INFO [M:0;10.0.0.110:51047] regionserver.HRegionServer(1363): Serving as 10.0.0.110,51047,1434564478492, RpcServer on 10.0.0.110/10.0.0.110:51047, sessionid=0x14e02b3a232001d 2015-06-17 11:07:58,617 DEBUG [M:0;10.0.0.110:51047] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2015-06-17 11:07:58,617 DEBUG [M:0;10.0.0.110:51047] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager 10.0.0.110,51047,1434564478492 2015-06-17 11:07:58,617 DEBUG [M:0;10.0.0.110:51047] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,51047,1434564478492' 2015-06-17 11:07:58,617 DEBUG [M:0;10.0.0.110:51047] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2015-06-17 11:07:58,618 DEBUG [M:0;10.0.0.110:51047] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2015-06-17 11:07:58,618 DEBUG [M:0;10.0.0.110:51047] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2015-06-17 11:07:58,618 DEBUG [M:0;10.0.0.110:51047] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2015-06-17 11:07:58,618 DEBUG [M:0;10.0.0.110:51047] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager 10.0.0.110,51047,1434564478492 2015-06-17 11:07:58,618 DEBUG [M:0;10.0.0.110:51047] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,51047,1434564478492' 2015-06-17 11:07:58,618 DEBUG [M:0;10.0.0.110:51047] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2015-06-17 11:07:58,618 DEBUG [M:0;10.0.0.110:51047] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2015-06-17 11:07:58,619 DEBUG [M:0;10.0.0.110:51047] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2015-06-17 11:07:58,619 INFO [M:0;10.0.0.110:51047] quotas.RegionServerQuotaManager(62): Quota support disabled 2015-06-17 11:07:58,622 INFO [10.0.0.110:51047.activeMasterManager] master.ServerManager(1028): Finished waiting for region servers count to settle; checked in 1, slept for 50 ms, expecting minimum of 1, maximum of 1, master is running 2015-06-17 11:07:58,623 INFO [10.0.0.110:51047.activeMasterManager] master.ServerManager(459): Registering server=10.0.0.110,50905,1434564404764 2015-06-17 11:07:58,623 INFO [10.0.0.110:51047.activeMasterManager] master.HMaster(665): Registered server found up in zk but who has not yet reported in: 10.0.0.110,50905,1434564404764 2015-06-17 11:07:58,625 INFO [10.0.0.110:51047.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50905,1434564404764 belongs to an existing region server 2015-06-17 11:07:58,626 INFO [10.0.0.110:51047.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51047,1434564478492 belongs to an existing region server 2015-06-17 11:07:58,628 WARN [10.0.0.110:51047.activeMasterManager] master.RegionStates(339): Open region's last host null should be the same as the current one 10.0.0.110,51015,1434564463167, ignored the last and used the current one 2015-06-17 11:07:58,730 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:07:58,731 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=350, started=15163 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:07:58,732 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:58,732 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51047: callId: 1016 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:58,733 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2538 ms ago, cancelled=false, msg= 2015-06-17 11:07:58,782 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=171, retries=350, started=48334 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:58,932 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=350, started=15364 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:07:58,934 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:58,934 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51047: callId: 1018 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:58,935 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2740 ms ago, cancelled=false, msg= 2015-06-17 11:07:58,984 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=172, retries=350, started=48536 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:59,132 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=350, started=15564 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:07:59,136 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:59,136 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51047: callId: 1020 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:59,136 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2941 ms ago, cancelled=false, msg= 2015-06-17 11:07:59,185 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=173, retries=350, started=48737 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:59,332 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=350, started=15764 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:07:59,339 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:59,339 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51047: callId: 1022 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:59,340 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3145 ms ago, cancelled=false, msg= 2015-06-17 11:07:59,386 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=174, retries=350, started=48938 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:59,533 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=350, started=15965 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:07:59,542 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:59,542 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51047: callId: 1024 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:59,543 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3348 ms ago, cancelled=false, msg= 2015-06-17 11:07:59,587 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=175, retries=350, started=49139 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:59,735 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=350, started=16167 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:07:59,745 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:59,745 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51047: callId: 1026 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:59,745 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3550 ms ago, cancelled=false, msg= 2015-06-17 11:07:59,789 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=176, retries=350, started=49341 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:07:59,936 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=350, started=16368 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:07:59,947 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:07:59,947 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51047: callId: 1028 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:07:59,948 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3753 ms ago, cancelled=false, msg= 2015-06-17 11:07:59,991 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=177, retries=350, started=49543 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:00,136 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=350, started=16568 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:00,150 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:00,150 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51047: callId: 1030 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:00,150 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3955 ms ago, cancelled=false, msg= 2015-06-17 11:08:00,192 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=178, retries=350, started=49744 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:00,337 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=350, started=16769 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:00,352 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:00,353 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51047: callId: 1032 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:00,353 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4158 ms ago, cancelled=false, msg= 2015-06-17 11:08:00,392 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=179, retries=350, started=49944 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:00,463 DEBUG [AsyncRpcChannel-pool2-t6] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2015-06-17 11:08:00,463 DEBUG [RpcServer.listener,port=51047] ipc.RpcServer$Listener(840): RpcServer.listener,port=51047: connection from 10.0.0.110:51057; # active connections: 2 2015-06-17 11:08:00,464 INFO [RpcServer.reader=2,bindAddress=10.0.0.110,port=51047] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51057 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:08:00,538 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=350, started=16969 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:00,555 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:00,555 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51047: callId: 1034 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:00,556 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4361 ms ago, cancelled=false, msg= 2015-06-17 11:08:00,593 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=180, retries=350, started=50145 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:00,738 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=350, started=17170 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:00,757 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:00,757 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51047: callId: 1036 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:00,757 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4562 ms ago, cancelled=false, msg= 2015-06-17 11:08:00,794 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=181, retries=350, started=50346 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:00,939 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=350, started=17371 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:00,959 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:00,960 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51047: callId: 1038 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:00,960 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4765 ms ago, cancelled=false, msg= 2015-06-17 11:08:00,996 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=182, retries=350, started=50548 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:01,140 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=350, started=17572 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:01,162 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:01,162 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51047: callId: 1040 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:01,162 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4967 ms ago, cancelled=false, msg= 2015-06-17 11:08:01,196 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=183, retries=350, started=50748 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:01,341 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=350, started=17773 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:01,365 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:01,365 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51047: callId: 1042 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:01,366 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5171 ms ago, cancelled=false, msg= 2015-06-17 11:08:01,397 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=184, retries=350, started=50949 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:01,543 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=350, started=17975 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:01,569 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:01,569 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51047: callId: 1044 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:01,569 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5374 ms ago, cancelled=false, msg= 2015-06-17 11:08:01,570 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564476195, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:08:01,598 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=185, retries=350, started=51150 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:01,744 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=350, started=18176 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:01,773 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:01,773 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51047: callId: 1046 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:01,776 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:01,776 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51047: callId: 1048 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:01,780 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:01,780 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51047: callId: 1050 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:01,786 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:01,787 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51047: callId: 1052 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:01,798 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:01,798 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51047: callId: 1054 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:01,800 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=186, retries=350, started=51352 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:01,819 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:01,819 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51047: callId: 1056 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:01,862 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:01,862 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51047: callId: 1058 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:01,945 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=350, started=18377 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:01,964 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:01,964 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51047: callId: 1060 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:02,001 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=187, retries=350, started=51553 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:02,065 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:02,066 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51047: callId: 1062 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:02,145 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=350, started=18577 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:02,167 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:02,167 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51047: callId: 1064 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:02,202 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=188, retries=350, started=51754 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:02,268 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:02,268 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51047: callId: 1066 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:02,269 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=497 ms ago, cancelled=false, msg= 2015-06-17 11:08:02,347 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=350, started=18779 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:02,403 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=189, retries=350, started=51955 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:02,470 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:02,470 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51047: callId: 1068 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:02,470 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=698 ms ago, cancelled=false, msg= 2015-06-17 11:08:02,548 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=350, started=18980 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:02,604 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=190, retries=350, started=52156 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:02,672 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:02,673 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51047: callId: 1070 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:02,673 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=901 ms ago, cancelled=false, msg= 2015-06-17 11:08:02,749 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=350, started=19181 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:02,804 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=191, retries=350, started=52356 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:02,874 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:02,875 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51047: callId: 1072 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:02,875 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1103 ms ago, cancelled=false, msg= 2015-06-17 11:08:02,949 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=350, started=19381 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:03,004 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=192, retries=350, started=52556 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:03,078 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:03,078 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51047: callId: 1074 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:03,079 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1307 ms ago, cancelled=false, msg= 2015-06-17 11:08:03,151 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=350, started=19583 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:03,206 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=193, retries=350, started=52758 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:03,281 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:03,281 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51047: callId: 1076 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:03,281 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1509 ms ago, cancelled=false, msg= 2015-06-17 11:08:03,352 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=350, started=19784 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:03,406 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=194, retries=350, started=52958 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:03,482 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:03,482 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51047: callId: 1078 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:03,483 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1711 ms ago, cancelled=false, msg= 2015-06-17 11:08:03,553 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=35, retries=350, started=19985 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:03,606 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=195, retries=350, started=53158 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:03,684 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:03,684 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51047: callId: 1080 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:03,685 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1913 ms ago, cancelled=false, msg= 2015-06-17 11:08:03,755 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=36, retries=350, started=20187 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:03,807 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=196, retries=350, started=53359 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:03,887 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:03,887 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51047: callId: 1082 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:03,887 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2115 ms ago, cancelled=false, msg= 2015-06-17 11:08:03,957 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=37, retries=350, started=20389 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:04,009 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=197, retries=350, started=53561 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:04,089 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:04,089 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51047: callId: 1084 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:04,089 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2317 ms ago, cancelled=false, msg= 2015-06-17 11:08:04,158 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=38, retries=350, started=20590 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:04,210 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=198, retries=350, started=53762 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:04,290 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:04,291 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51047: callId: 1086 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:04,291 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2519 ms ago, cancelled=false, msg= 2015-06-17 11:08:04,359 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=39, retries=350, started=20791 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:04,411 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=199, retries=350, started=53963 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:04,492 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:04,492 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51047: callId: 1088 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:04,493 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2721 ms ago, cancelled=false, msg= 2015-06-17 11:08:04,561 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=40, retries=350, started=20993 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:04,611 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=200, retries=350, started=54163 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:04,695 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:04,695 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51047: callId: 1090 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:04,695 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2923 ms ago, cancelled=false, msg= 2015-06-17 11:08:04,761 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=41, retries=350, started=21193 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:04,813 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=201, retries=350, started=54365 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:04,897 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:04,897 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51047: callId: 1092 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:04,898 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3126 ms ago, cancelled=false, msg= 2015-06-17 11:08:04,962 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=42, retries=350, started=21394 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:05,015 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=202, retries=350, started=54567 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:05,100 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:05,100 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51047: callId: 1094 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:05,100 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3328 ms ago, cancelled=false, msg= 2015-06-17 11:08:05,164 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=43, retries=350, started=21596 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:05,216 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=203, retries=350, started=54768 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:05,302 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:05,302 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51047: callId: 1096 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:05,302 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3530 ms ago, cancelled=false, msg= 2015-06-17 11:08:05,365 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=44, retries=350, started=21797 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:05,417 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=204, retries=350, started=54969 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:05,506 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:05,506 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51047: callId: 1098 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:05,507 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3735 ms ago, cancelled=false, msg= 2015-06-17 11:08:05,567 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=45, retries=350, started=21999 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:05,618 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=205, retries=350, started=55170 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:05,709 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:05,710 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51047: callId: 1100 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:05,710 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3938 ms ago, cancelled=false, msg= 2015-06-17 11:08:05,768 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=46, retries=350, started=22200 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:05,820 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=206, retries=350, started=55372 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:05,911 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:05,911 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51047: callId: 1102 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:05,912 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4140 ms ago, cancelled=false, msg= 2015-06-17 11:08:05,968 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=47, retries=350, started=22400 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:06,021 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=207, retries=350, started=55573 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:06,113 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:06,113 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51047: callId: 1104 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:06,113 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4341 ms ago, cancelled=false, msg= 2015-06-17 11:08:06,169 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=48, retries=350, started=22601 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:06,221 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=208, retries=350, started=55773 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:06,315 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:06,315 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51047: callId: 1106 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:06,315 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4543 ms ago, cancelled=false, msg= 2015-06-17 11:08:06,370 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=49, retries=350, started=22802 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:06,422 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=209, retries=350, started=55974 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:06,517 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:06,517 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51047: callId: 1108 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:06,518 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4746 ms ago, cancelled=false, msg= 2015-06-17 11:08:06,571 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=50, retries=350, started=23003 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:06,624 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=210, retries=350, started=56176 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:06,719 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:06,719 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51047: callId: 1110 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:06,719 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4947 ms ago, cancelled=false, msg= 2015-06-17 11:08:06,773 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=51, retries=350, started=23205 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:06,824 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=211, retries=350, started=56376 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:06,920 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:06,921 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51047: callId: 1112 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:06,921 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5149 ms ago, cancelled=false, msg= 2015-06-17 11:08:06,975 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=52, retries=350, started=23407 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:07,024 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=212, retries=350, started=56576 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:07,122 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:07,122 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51047: callId: 1114 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:07,123 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5351 ms ago, cancelled=false, msg= 2015-06-17 11:08:07,123 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564481772, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:08:07,176 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=53, retries=350, started=23608 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:07,224 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=213, retries=350, started=56776 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:07,325 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:07,326 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51047: callId: 1116 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:07,329 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:07,329 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51047: callId: 1118 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:07,334 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:07,334 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51047: callId: 1120 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:07,341 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:07,341 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51047: callId: 1122 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:07,353 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:07,353 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51047: callId: 1124 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:07,375 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:07,375 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51047: callId: 1126 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:07,376 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=54, retries=350, started=23808 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:07,417 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:07,417 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51047: callId: 1128 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:07,426 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=214, retries=350, started=56978 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:07,520 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:07,520 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51047: callId: 1130 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:07,576 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=55, retries=350, started=24008 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:07,621 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:07,621 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51047: callId: 1132 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:07,627 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=215, retries=350, started=57179 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:07,723 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:07,724 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51047: callId: 1134 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:07,777 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=56, retries=350, started=24209 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:07,826 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:07,826 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51047: callId: 1136 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:07,826 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=502 ms ago, cancelled=false, msg= 2015-06-17 11:08:07,828 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=216, retries=350, started=57380 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:07,978 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=57, retries=350, started=24410 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:08,027 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:08,028 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51047: callId: 1138 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:08,028 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=704 ms ago, cancelled=false, msg= 2015-06-17 11:08:08,029 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=217, retries=350, started=57581 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:08,180 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=58, retries=350, started=24612 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:08,230 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:08,230 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51047: callId: 1140 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:08,230 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=218, retries=350, started=57782 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:08,230 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=906 ms ago, cancelled=false, msg= 2015-06-17 11:08:08,381 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=59, retries=350, started=24813 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:08,431 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=219, retries=350, started=57983 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:08,432 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:08,432 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51047: callId: 1142 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:08,433 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1109 ms ago, cancelled=false, msg= 2015-06-17 11:08:08,537 DEBUG [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(188): Sleeping for 3500 before executing next master restart count=5 2015-06-17 11:08:08,582 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=60, retries=350, started=25014 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:08,631 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=220, retries=350, started=58183 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:08,635 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:08,635 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51047: callId: 1144 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:08,635 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1311 ms ago, cancelled=false, msg= 2015-06-17 11:08:08,783 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=61, retries=350, started=25215 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:08,832 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=221, retries=350, started=58384 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:08,837 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:08,838 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51047: callId: 1146 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:08,838 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1514 ms ago, cancelled=false, msg= 2015-06-17 11:08:08,983 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=62, retries=350, started=25415 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:09,035 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=222, retries=350, started=58587 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:09,039 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:09,039 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51047: callId: 1148 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:09,040 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1716 ms ago, cancelled=false, msg= 2015-06-17 11:08:09,183 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=63, retries=350, started=25615 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:09,235 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=223, retries=350, started=58787 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:09,242 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:09,242 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51047: callId: 1150 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:09,242 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1918 ms ago, cancelled=false, msg= 2015-06-17 11:08:09,384 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=64, retries=350, started=25816 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:09,437 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=224, retries=350, started=58989 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:09,445 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:09,445 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51047: callId: 1152 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:09,445 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2121 ms ago, cancelled=false, msg= 2015-06-17 11:08:09,586 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=65, retries=350, started=26018 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:09,638 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=225, retries=350, started=59190 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:09,647 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:09,647 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51047: callId: 1154 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:09,648 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2324 ms ago, cancelled=false, msg= 2015-06-17 11:08:09,788 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=66, retries=350, started=26220 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:09,840 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=226, retries=350, started=59392 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:09,850 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:09,850 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51047: callId: 1156 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:09,850 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2526 ms ago, cancelled=false, msg= 2015-06-17 11:08:09,988 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=67, retries=350, started=26420 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:10,040 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=227, retries=350, started=59592 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:10,052 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:10,052 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51047: callId: 1158 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:10,053 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2729 ms ago, cancelled=false, msg= 2015-06-17 11:08:10,189 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=68, retries=350, started=26621 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:10,240 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=228, retries=350, started=59792 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:10,255 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:10,255 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51047: callId: 1160 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:10,255 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2931 ms ago, cancelled=false, msg= 2015-06-17 11:08:10,390 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=69, retries=350, started=26822 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:10,442 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=229, retries=350, started=59994 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:10,457 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:10,457 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51047: callId: 1162 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:10,457 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3133 ms ago, cancelled=false, msg= 2015-06-17 11:08:10,591 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=70, retries=350, started=27023 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:10,642 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=230, retries=350, started=60194 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:10,660 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:10,660 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51047: callId: 1164 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:10,661 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3337 ms ago, cancelled=false, msg= 2015-06-17 11:08:10,792 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=71, retries=350, started=27224 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:10,844 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=231, retries=350, started=60395 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:10,862 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:10,862 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51047: callId: 1166 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:10,862 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3538 ms ago, cancelled=false, msg= 2015-06-17 11:08:10,993 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=72, retries=350, started=27425 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:11,044 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=232, retries=350, started=60596 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:11,065 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:11,065 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51047: callId: 1168 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:11,065 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3741 ms ago, cancelled=false, msg= 2015-06-17 11:08:11,195 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=73, retries=350, started=27627 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:11,246 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=233, retries=350, started=60798 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:11,266 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:11,266 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51047: callId: 1170 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:11,267 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3943 ms ago, cancelled=false, msg= 2015-06-17 11:08:11,395 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=74, retries=350, started=27827 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:11,447 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=234, retries=350, started=60999 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:11,469 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:11,469 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51047: callId: 1172 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:11,470 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4146 ms ago, cancelled=false, msg= 2015-06-17 11:08:11,596 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=75, retries=350, started=28028 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:11,647 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=235, retries=350, started=61199 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:11,672 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:11,672 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51047: callId: 1174 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:11,673 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4349 ms ago, cancelled=false, msg= 2015-06-17 11:08:11,797 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=76, retries=350, started=28229 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:11,848 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=236, retries=350, started=61400 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:11,874 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:11,875 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51047: callId: 1176 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:11,875 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4551 ms ago, cancelled=false, msg= 2015-06-17 11:08:11,998 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=77, retries=350, started=28429 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:12,037 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(197): executing master kill 2015-06-17 11:08:12,037 INFO [MasterRestartThread] hbase.MiniHBaseCluster(422): Aborting 10.0.0.110,51047,1434564478492 2015-06-17 11:08:12,038 FATAL [MasterRestartThread] master.HMaster(1953): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2015-06-17 11:08:12,038 FATAL [MasterRestartThread] master.HMaster(1956): Aborting for tests java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:423) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:270) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$5.run(TestMasterRestartWithProcedures.java:198) 2015-06-17 11:08:12,038 INFO [MasterRestartThread] regionserver.HRegionServer(1859): STOPPED: Aborting for tests 2015-06-17 11:08:12,038 INFO [M:0;10.0.0.110:51047] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2015-06-17 11:08:12,038 INFO [MasterRestartThread] hbase.LocalHBaseCluster(379): Waiting on 10.0.0.110,51047,1434564478492 2015-06-17 11:08:12,038 INFO [SplitLogWorker-10.0.0.110:51047] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2015-06-17 11:08:12,038 INFO [M:0;10.0.0.110:51047] regionserver.HeapMemoryManager(201): Stoping HeapMemoryTuner chore. 2015-06-17 11:08:12,038 INFO [SplitLogWorker-10.0.0.110:51047] regionserver.SplitLogWorker(155): SplitLogWorker 10.0.0.110,51047,1434564478492 exiting 2015-06-17 11:08:12,038 INFO [M:0;10.0.0.110:51047] procedure2.ProcedureExecutor(490): Stopping the procedure executor 2015-06-17 11:08:12,039 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2015-06-17 11:08:12,038 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2015-06-17 11:08:12,040 INFO [M:0;10.0.0.110:51047] wal.WALProcedureStore(159): Stopping the WAL Procedure Store 2015-06-17 11:08:12,047 INFO [IPC Server handler 2 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741872_1048{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 0 2015-06-17 11:08:12,048 INFO [M:0;10.0.0.110:51047] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2015-06-17 11:08:12,048 INFO [M:0;10.0.0.110:51047] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2015-06-17 11:08:12,048 INFO [M:0;10.0.0.110:51047] regionserver.HRegionServer(1021): stopping server 10.0.0.110,51047,1434564478492 2015-06-17 11:08:12,048 DEBUG [M:0;10.0.0.110:51047] zookeeper.MetaTableLocator(617): Stopping MetaTableLocator 2015-06-17 11:08:12,048 INFO [M:0;10.0.0.110:51047] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a232001e 2015-06-17 11:08:12,048 DEBUG [M:0;10.0.0.110:51047] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:08:12,049 INFO [M:0;10.0.0.110:51047] regionserver.HRegionServer(1049): stopping server 10.0.0.110,51047,1434564478492; all regions closed. 2015-06-17 11:08:12,049 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=237, retries=350, started=61601 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:12,049 DEBUG [M:0;10.0.0.110:51047] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51047,1434564478492 2015-06-17 11:08:12,049 INFO [10.0.0.110:51047.activeMasterManager] zookeeper.MetaTableLocator(340): Failed verification of hbase:meta,,1 at address=10.0.0.110,51015,1434564463167, exception=Call id=0 on server /10.0.0.110:51015 aborted: connection is closing 2015-06-17 11:08:12,049 DEBUG [AsyncRpcChannel-pool2-t5] ipc.AsyncRpcChannel$8(574): IPC Client (1209434512) connection to /10.0.0.110:51015 from ndimiduk.hfs.5: closed 2015-06-17 11:08:12,050 DEBUG [10.0.0.110:51047.activeMasterManager] zookeeper.ZKUtil(745): master:51047-0x14e02b3a232001d, quorum=localhost:51848, baseZNode=/hbase Unable to get data of znode /hbase/recovering-regions/1588230740 because node does not exist (not an error) 2015-06-17 11:08:12,052 INFO [IPC Server handler 6 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741871_1047{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 83 2015-06-17 11:08:12,053 DEBUG [10.0.0.110:51047.activeMasterManager] coordination.ZKSplitLogManagerCoordination(650): Marked 1588230740 recovering from 10.0.0.110,51015,1434564463167: /hbase/recovering-regions/1588230740/10.0.0.110,51015,1434564463167 2015-06-17 11:08:12,053 INFO [10.0.0.110:51047.activeMasterManager] master.HMaster(884): Re-assigning hbase:meta with replicaId, 0 it was on 10.0.0.110,51015,1434564463167 2015-06-17 11:08:12,053 INFO [10.0.0.110:51047.activeMasterManager] master.RegionStates(1072): Transition {1588230740 state=OPEN, ts=1434564478628, server=10.0.0.110,51015,1434564463167} to {1588230740 state=OFFLINE, ts=1434564492053, server=10.0.0.110,51015,1434564463167} 2015-06-17 11:08:12,053 INFO [10.0.0.110:51047.activeMasterManager] zookeeper.MetaTableLocator(440): Setting hbase:meta region location in ZooKeeper as 10.0.0.110,51015,1434564463167 2015-06-17 11:08:12,055 INFO [10.0.0.110:51047.activeMasterManager] master.AssignmentManager(996): Skip assigning hbase:meta,,1.1588230740, the server is stopped/aborted 2015-06-17 11:08:12,055 FATAL [10.0.0.110:51047.activeMasterManager] master.HMaster$1(1546): Failed to become active master java.io.IOException: connection is closed at org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:254) at org.apache.hadoop.hbase.MetaTableAccessor.putToMetaTable(MetaTableAccessor.java:1341) at org.apache.hadoop.hbase.MetaTableAccessor.updateTableState(MetaTableAccessor.java:1691) at org.apache.hadoop.hbase.MetaTableAccessor.updateTableState(MetaTableAccessor.java:1110) at org.apache.hadoop.hbase.master.TableStateManager.udpateMetaState(TableStateManager.java:180) at org.apache.hadoop.hbase.master.TableStateManager.setTableState(TableStateManager.java:67) at org.apache.hadoop.hbase.master.HMaster.assignMeta(HMaster.java:892) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:711) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1542) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:12,058 DEBUG [M:0;10.0.0.110:51047] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:08:12,058 INFO [M:0;10.0.0.110:51047] wal.FSHLog(977): Closed WAL: FSHLog 10.0.0.110%2C51047%2C1434564478492.default:(num 1434564478578) 2015-06-17 11:08:12,058 DEBUG [M:0;10.0.0.110:51047] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:08:12,058 INFO [M:0;10.0.0.110:51047] regionserver.Leases(146): M:0;10.0.0.110:51047 closing leases 2015-06-17 11:08:12,058 INFO [M:0;10.0.0.110:51047] regionserver.Leases(149): M:0;10.0.0.110:51047 closed leases 2015-06-17 11:08:12,058 INFO [M:0;10.0.0.110:51047] hbase.ChoreService(302): Chore service for: 10.0.0.110,51047,1434564478492 had [[ScheduledChore: Name: MovedRegionsCleaner for region 10.0.0.110,51047,1434564478492 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.0.0.110,51047,1434564478492-MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS]] on shutdown 2015-06-17 11:08:12,078 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:12,078 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51047: callId: 1178 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:12,078 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4754 ms ago, cancelled=false, msg= 2015-06-17 11:08:12,199 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=78, retries=350, started=28631 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:12,250 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=238, retries=350, started=61802 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:12,282 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:12,282 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51047: callId: 1180 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:12,283 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4959 ms ago, cancelled=false, msg= 2015-06-17 11:08:12,401 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=79, retries=350, started=28833 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:12,451 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=239, retries=350, started=62003 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:12,489 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:12,489 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51047] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51047: callId: 1182 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51051 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:12,490 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5166 ms ago, cancelled=false, msg= 2015-06-17 11:08:12,530 INFO [10.0.0.110,51047,1434564478492_splitLogManager__ChoreService_1] hbase.ScheduledChore(179): Chore: SplitLogManager Timeout Monitor was stopped 2015-06-17 11:08:12,590 INFO [master//10.0.0.110:0.logRoller] regionserver.LogRoller(159): LogRoller exiting. 2015-06-17 11:08:12,590 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(146): master//10.0.0.110:0.leaseChecker closing leases 2015-06-17 11:08:12,590 INFO [M:0;10.0.0.110:51047] regionserver.CompactSplitThread(394): Waiting for Split Thread to finish... 2015-06-17 11:08:12,591 INFO [M:0;10.0.0.110:51047] regionserver.CompactSplitThread(394): Waiting for Merge Thread to finish... 2015-06-17 11:08:12,591 INFO [M:0;10.0.0.110:51047] regionserver.CompactSplitThread(394): Waiting for Large Compaction Thread to finish... 2015-06-17 11:08:12,591 INFO [M:0;10.0.0.110:51047] regionserver.CompactSplitThread(394): Waiting for Small Compaction Thread to finish... 2015-06-17 11:08:12,590 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(149): master//10.0.0.110:0.leaseChecker closed leases 2015-06-17 11:08:12,595 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51047-0x14e02b3a232001d, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/10.0.0.110,51047,1434564478492 2015-06-17 11:08:12,595 INFO [M:0;10.0.0.110:51047] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320021 2015-06-17 11:08:12,596 DEBUG [M:0;10.0.0.110:51047] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:08:12,597 DEBUG [M:0;10.0.0.110:51047] master.HMaster(1074): Stopping service threads 2015-06-17 11:08:12,598 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51047-0x14e02b3a232001d, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:08:12,598 INFO [M:0;10.0.0.110:51047] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a232001f 2015-06-17 11:08:12,598 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:08:12,598 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(488): master:51047-0x14e02b3a232001d, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:08:12,598 DEBUG [M:0;10.0.0.110:51047] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:08:12,598 DEBUG [main-EventThread] zookeeper.ZKUtil(488): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:08:12,599 INFO [M:0;10.0.0.110:51047] hbase.ChoreService(302): Chore service for: 10.0.0.110,51047,1434564478492_splitLogManager_ had [] on shutdown 2015-06-17 11:08:12,599 INFO [M:0;10.0.0.110:51047] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2015-06-17 11:08:12,599 INFO [M:0;10.0.0.110:51047] ipc.RpcServer(2219): Stopping server on 51047 2015-06-17 11:08:12,599 INFO [RpcServer.listener,port=51047] ipc.RpcServer$Listener(776): RpcServer.listener,port=51047: stopping 2015-06-17 11:08:12,599 INFO [RpcServer.responder] ipc.RpcServer$Responder(1018): RpcServer.responder: stopped 2015-06-17 11:08:12,599 INFO [RpcServer.responder] ipc.RpcServer$Responder(921): RpcServer.responder: stopping 2015-06-17 11:08:12,600 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51047-0x14e02b3a232001d, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,51047,1434564478492 2015-06-17 11:08:12,600 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,51047,1434564478492 2015-06-17 11:08:12,600 INFO [MasterRestartThread-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [10.0.0.110,51047,1434564478492] 2015-06-17 11:08:12,601 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51047-0x14e02b3a232001d, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:08:12,601 INFO [M:0;10.0.0.110:51047] regionserver.HRegionServer(1093): stopping server 10.0.0.110,51047,1434564478492; zookeeper connection closed. 2015-06-17 11:08:12,601 INFO [M:0;10.0.0.110:51047] regionserver.HRegionServer(1096): M:0;10.0.0.110:51047 exiting 2015-06-17 11:08:12,601 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(200): executing master restart 2015-06-17 11:08:12,601 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=80, retries=350, started=29033 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:12,602 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:12,602 INFO [main-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(121): /hbase/rs/10.0.0.110,51047,1434564478492 znode expired, triggering replicatorRemoved event 2015-06-17 11:08:12,602 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:08:12,602 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:12,615 INFO [MasterRestartThread] client.ConnectionUtils(105): master//10.0.0.110:0 server-side HConnection retries=350 2015-06-17 11:08:12,616 INFO [MasterRestartThread] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2015-06-17 11:08:12,617 INFO [MasterRestartThread] ipc.RpcServer$Listener(598): master//10.0.0.110:0: started 10 reader(s). 2015-06-17 11:08:12,619 INFO [MasterRestartThread] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:08:12,620 INFO [MasterRestartThread] zookeeper.RecoverableZooKeeper(121): Process identifier=master:51074 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:08:12,622 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:510740x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:08:12,623 DEBUG [MasterRestartThread] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-17 11:08:12,624 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(578): master:51074-0x14e02b3a2320022 connected 2015-06-17 11:08:12,625 DEBUG [MasterRestartThread] zookeeper.ZKUtil(488): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:08:12,625 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2015-06-17 11:08:12,626 INFO [RpcServer.responder] ipc.RpcServer$Responder(917): RpcServer.responder: starting 2015-06-17 11:08:12,626 INFO [RpcServer.listener,port=51074] ipc.RpcServer$Listener(729): RpcServer.listener,port=51074: starting 2015-06-17 11:08:12,626 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=0 queue=0 2015-06-17 11:08:12,626 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=1 queue=0 2015-06-17 11:08:12,626 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=2 queue=0 2015-06-17 11:08:12,626 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=3 queue=0 2015-06-17 11:08:12,626 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=4 queue=0 2015-06-17 11:08:12,626 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=0 queue=0 2015-06-17 11:08:12,626 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=1 queue=1 2015-06-17 11:08:12,626 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=2 queue=0 2015-06-17 11:08:12,627 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=3 queue=1 2015-06-17 11:08:12,627 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=4 queue=0 2015-06-17 11:08:12,627 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=5 queue=1 2015-06-17 11:08:12,627 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=6 queue=0 2015-06-17 11:08:12,627 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=7 queue=1 2015-06-17 11:08:12,627 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=8 queue=0 2015-06-17 11:08:12,627 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=9 queue=1 2015-06-17 11:08:12,628 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=10 queue=0 2015-06-17 11:08:12,628 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=11 queue=1 2015-06-17 11:08:12,628 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=12 queue=0 2015-06-17 11:08:12,628 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=13 queue=1 2015-06-17 11:08:12,628 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=14 queue=0 2015-06-17 11:08:12,628 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=15 queue=1 2015-06-17 11:08:12,628 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=16 queue=0 2015-06-17 11:08:12,628 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=17 queue=1 2015-06-17 11:08:12,628 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=18 queue=0 2015-06-17 11:08:12,629 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=19 queue=1 2015-06-17 11:08:12,629 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=0 queue=0 2015-06-17 11:08:12,629 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=1 queue=0 2015-06-17 11:08:12,629 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=2 queue=0 2015-06-17 11:08:12,629 INFO [MasterRestartThread] master.HMaster(349): hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034, hbase.cluster.distributed=false 2015-06-17 11:08:12,629 INFO [MasterRestartThread] master.HMaster(1512): Adding backup master ZNode /hbase/backup-masters/10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,630 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,631 INFO [M:0;10.0.0.110:51074] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x3ea86006 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:08:12,631 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:08:12,631 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:08:12,632 DEBUG [10.0.0.110:51074.activeMasterManager] zookeeper.ZKUtil(486): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:08:12,632 INFO [10.0.0.110:51074.activeMasterManager] master.ActiveMasterManager(170): Deleting ZNode for /hbase/backup-masters/10.0.0.110,51074,1434564492618 from backup master directory 2015-06-17 11:08:12,632 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(486): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:08:12,632 DEBUG [MasterRestartThread-EventThread] master.ActiveMasterManager(126): A master is now available 2015-06-17 11:08:12,633 DEBUG [M:0;10.0.0.110:51074-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x3ea860060x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:08:12,633 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,633 WARN [10.0.0.110:51074.activeMasterManager] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:08:12,633 INFO [10.0.0.110:51074.activeMasterManager] master.ActiveMasterManager(179): Registered Active Master=10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,633 DEBUG [M:0;10.0.0.110:51074] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a2564a8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:08:12,633 DEBUG [M:0;10.0.0.110:51074] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:08:12,634 DEBUG [M:0;10.0.0.110:51074] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:08:12,634 DEBUG [M:0;10.0.0.110:51074-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x3ea86006-0x14e02b3a2320023 connected 2015-06-17 11:08:12,635 INFO [M:0;10.0.0.110:51074] regionserver.HRegionServer(774): ClusterId : b7c4b21b-aab2-44b4-8562-2b61af22ad6c 2015-06-17 11:08:12,635 DEBUG [M:0;10.0.0.110:51074] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2015-06-17 11:08:12,635 DEBUG [M:0;10.0.0.110:51074] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:08:12,636 DEBUG [M:0;10.0.0.110:51074] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2015-06-17 11:08:12,636 DEBUG [M:0;10.0.0.110:51074] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2015-06-17 11:08:12,636 DEBUG [M:0;10.0.0.110:51074] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:08:12,637 DEBUG [M:0;10.0.0.110:51074] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2015-06-17 11:08:12,637 INFO [M:0;10.0.0.110:51074] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2015-06-17 11:08:12,638 INFO [M:0;10.0.0.110:51074] compactions.PressureAwareCompactionThroughputController(175): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2015-06-17 11:08:12,638 INFO [M:0;10.0.0.110:51074] regionserver.HRegionServer$CompactionChecker(1504): CompactionChecker runs every 1sec 2015-06-17 11:08:12,638 DEBUG [M:0;10.0.0.110:51074] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@573814ce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=10.0.0.110/10.0.0.110:0 2015-06-17 11:08:12,638 DEBUG [M:0;10.0.0.110:51074] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:08:12,639 DEBUG [M:0;10.0.0.110:51074] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:08:12,639 DEBUG [M:0;10.0.0.110:51074] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:M:0;10.0.0.110:51074 2015-06-17 11:08:12,639 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:08:12,640 DEBUG [M:0;10.0.0.110:51074] zookeeper.ZKUtil(486): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,640 INFO [M:0;10.0.0.110:51074] regionserver.RegionServerCoprocessorHost(65): System coprocessor loading is enabled 2015-06-17 11:08:12,640 INFO [M:0;10.0.0.110:51074] regionserver.RegionServerCoprocessorHost(66): Table coprocessor loading is enabled 2015-06-17 11:08:12,640 INFO [M:0;10.0.0.110:51074] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,51074,1434564492618 with port=51074, startcode=1434564492618 2015-06-17 11:08:12,640 DEBUG [M:0;10.0.0.110:51074] regionserver.HRegionServer(2295): Master is not running yet 2015-06-17 11:08:12,640 WARN [M:0;10.0.0.110:51074] regionserver.HRegionServer(900): reportForDuty failed; sleeping and then retrying. 2015-06-17 11:08:12,640 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:12,640 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,645 DEBUG [10.0.0.110:51074.activeMasterManager] util.FSTableDescriptors(807): Current tableInfoPath = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2015-06-17 11:08:12,647 DEBUG [10.0.0.110:51074.activeMasterManager] util.FSTableDescriptors(811): TableInfo already exists.. Skipping creation 2015-06-17 11:08:12,650 INFO [10.0.0.110:51074.activeMasterManager] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:08:12,650 DEBUG [10.0.0.110:51074.activeMasterManager] coordination.ZKSplitLogManagerCoordination(885): Distributed log replay=true 2015-06-17 11:08:12,651 INFO [10.0.0.110:51074.activeMasterManager] coordination.ZKSplitLogManagerCoordination(599): Found 0 orphan tasks and 0 rescan nodes 2015-06-17 11:08:12,651 DEBUG [10.0.0.110:51074.activeMasterManager] util.FSTableDescriptors(222): Fetching table descriptors from the filesystem. 2015-06-17 11:08:12,652 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=240, retries=350, started=62204 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:12,661 INFO [10.0.0.110:51074.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x6717a620 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:08:12,663 DEBUG [10.0.0.110:51074.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x6717a6200x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:08:12,664 DEBUG [10.0.0.110:51074.activeMasterManager] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ef6e38f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:08:12,664 DEBUG [10.0.0.110:51074.activeMasterManager] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:08:12,664 DEBUG [10.0.0.110:51074.activeMasterManager] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:08:12,665 DEBUG [10.0.0.110:51074.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x6717a620-0x14e02b3a2320024 connected 2015-06-17 11:08:12,665 INFO [10.0.0.110:51074.activeMasterManager] balancer.StochasticLoadBalancer(135): loading config 2015-06-17 11:08:12,666 DEBUG [10.0.0.110:51074.activeMasterManager] zookeeper.ZKUtil(488): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2015-06-17 11:08:12,666 DEBUG [10.0.0.110:51074.activeMasterManager] zookeeper.ZKUtil(486): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:12,667 DEBUG [10.0.0.110:51074.activeMasterManager] zookeeper.ZKUtil(486): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,667 DEBUG [10.0.0.110:51074.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:12,668 DEBUG [10.0.0.110:51074.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,668 INFO [10.0.0.110:51074.activeMasterManager] master.HMaster(568): Server active/primary master=10.0.0.110,51074,1434564492618, sessionid=0x14e02b3a2320022, setting cluster-up flag (Was=true) 2015-06-17 11:08:12,669 DEBUG [10.0.0.110:51074.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:08:12,670 INFO [10.0.0.110:51074.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2015-06-17 11:08:12,671 DEBUG [10.0.0.110:51074.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,672 DEBUG [10.0.0.110:51074.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:08:12,673 INFO [10.0.0.110:51074.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2015-06-17 11:08:12,673 DEBUG [10.0.0.110:51074.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,673 INFO [10.0.0.110:51074.activeMasterManager] master.MasterCoprocessorHost(86): System coprocessor loading is enabled 2015-06-17 11:08:12,674 DEBUG [10.0.0.110:51074.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_OPEN_REGION-10.0.0.110:51074, corePoolSize=5, maxPoolSize=5 2015-06-17 11:08:12,674 DEBUG [10.0.0.110:51074.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_CLOSE_REGION-10.0.0.110:51074, corePoolSize=5, maxPoolSize=5 2015-06-17 11:08:12,674 DEBUG [10.0.0.110:51074.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_SERVER_OPERATIONS-10.0.0.110:51074, corePoolSize=5, maxPoolSize=5 2015-06-17 11:08:12,674 DEBUG [10.0.0.110:51074.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_META_SERVER_OPERATIONS-10.0.0.110:51074, corePoolSize=5, maxPoolSize=5 2015-06-17 11:08:12,674 DEBUG [10.0.0.110:51074.activeMasterManager] executor.ExecutorService(91): Starting executor service name=M_LOG_REPLAY_OPS-10.0.0.110:51074, corePoolSize=10, maxPoolSize=10 2015-06-17 11:08:12,674 DEBUG [10.0.0.110:51074.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_TABLE_OPERATIONS-10.0.0.110:51074, corePoolSize=1, maxPoolSize=1 2015-06-17 11:08:12,674 INFO [10.0.0.110:51074.activeMasterManager] procedure2.ProcedureExecutor(446): Starting procedure executor threads=9 2015-06-17 11:08:12,674 INFO [10.0.0.110:51074.activeMasterManager] wal.WALProcedureStore(214): Starting WAL Procedure Store lease recovery 2015-06-17 11:08:12,675 INFO [10.0.0.110:51074.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log 2015-06-17 11:08:12,676 INFO [10.0.0.110:51074.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log after 1ms 2015-06-17 11:08:12,676 DEBUG [10.0.0.110:51074.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log; isDirectory=false; length=14915; replication=1; blocksize=134217728; modification_time=1434564413352; access_time=1434564405122; owner=ndimiduk; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:08:12,678 INFO [10.0.0.110:51074.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000006.log 2015-06-17 11:08:12,679 INFO [10.0.0.110:51074.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000006.log after 1ms 2015-06-17 11:08:12,679 DEBUG [10.0.0.110:51074.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000006.log; isDirectory=false; length=60; replication=1; blocksize=134217728; modification_time=1434564492047; access_time=1434564478559; owner=ndimiduk.hfs.5; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:08:12,683 DEBUG [10.0.0.110:51074.activeMasterManager] wal.WALProcedureStore(610): Roll new state log: 7 2015-06-17 11:08:12,684 INFO [10.0.0.110:51074.activeMasterManager] wal.WALProcedureStore(237): Lease acquired for flushLogId: 7 2015-06-17 11:08:12,684 INFO [10.0.0.110:51074.activeMasterManager] wal.ProcedureWALFormatReader(150): No active entry found in state log hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000006.log. removing it 2015-06-17 11:08:12,685 DEBUG [10.0.0.110:51074.activeMasterManager] procedure2.ProcedureExecutor$1(278): load procedures maxProcId=7 2015-06-17 11:08:12,687 DEBUG [10.0.0.110:51074.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=RUNNABLE isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=7 state=RUNNABLE 2015-06-17 11:08:12,687 DEBUG [10.0.0.110:51074.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DeleteTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=6 state=FINISHED 2015-06-17 11:08:12,687 DEBUG [10.0.0.110:51074.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:12,687 DEBUG [10.0.0.110:51074.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=5 state=FINISHED 2015-06-17 11:08:12,688 DEBUG [10.0.0.110:51074.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:12,688 DEBUG [10.0.0.110:51074.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: EnableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=4 state=FINISHED 2015-06-17 11:08:12,688 DEBUG [10.0.0.110:51074.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:12,688 DEBUG [10.0.0.110:51074.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=3 state=FINISHED 2015-06-17 11:08:12,688 DEBUG [10.0.0.110:51074.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:12,688 DEBUG [10.0.0.110:51074.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=2 state=FINISHED 2015-06-17 11:08:12,688 DEBUG [10.0.0.110:51074.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:12,688 DEBUG [10.0.0.110:51074.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=hbase:namespace) user=ndimiduk (auth:SIMPLE) id=1 state=FINISHED 2015-06-17 11:08:12,688 DEBUG [10.0.0.110:51074.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:12,688 DEBUG [10.0.0.110:51074.activeMasterManager] wal.WALProcedureStore(652): Remove log: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000006.log 2015-06-17 11:08:12,689 INFO [IPC Server handler 4 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741872_1048 127.0.0.1:50896 2015-06-17 11:08:12,691 DEBUG [10.0.0.110:51074.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2015-06-17 11:08:12,691 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:08:12,691 INFO [10.0.0.110:51074.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:08:12,693 DEBUG [AsyncRpcChannel-pool2-t7] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service MasterService, sasl=false 2015-06-17 11:08:12,693 DEBUG [RpcServer.listener,port=51074] ipc.RpcServer$Listener(840): RpcServer.listener,port=51074: connection from 10.0.0.110:51083; # active connections: 1 2015-06-17 11:08:12,693 DEBUG [10.0.0.110:51074.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:08:12,694 INFO [RpcServer.reader=1,bindAddress=10.0.0.110,port=51074] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51083 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:08:12,694 DEBUG [10.0.0.110:51074.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/replication/rs already exists 2015-06-17 11:08:12,694 DEBUG [10.0.0.110:51074.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2015-06-17 11:08:12,694 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51074: callId: 1184 service: MasterService methodName: IsMasterRunning size: 24 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:12,694 DEBUG [10.0.0.110:51074.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): replicationLogCleaner-0x14e02b3a2320025 connected 2015-06-17 11:08:12,694 DEBUG [10.0.0.110:51074.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner 2015-06-17 11:08:12,695 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5371 ms ago, cancelled=false, msg= 2015-06-17 11:08:12,695 DEBUG [10.0.0.110:51074.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2015-06-17 11:08:12,695 DEBUG [10.0.0.110:51074.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2015-06-17 11:08:12,695 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564487324, pause=1, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.ConnectionImplementation$StubMaker.makeStub(ConnectionImplementation.java:1222) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionImplementation.java:1242) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1393) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) ... 19 more Caused by: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionImplementation.java:1253) at org.apache.hadoop.hbase.client.ConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionImplementation.java:1191) at org.apache.hadoop.hbase.client.ConnectionImplementation$StubMaker.makeStub(ConnectionImplementation.java:1213) ... 23 more Caused by: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ipc.ServerNotRunningYetException): org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:08:12,695 DEBUG [10.0.0.110:51074.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2015-06-17 11:08:12,696 INFO [10.0.0.110:51074.activeMasterManager] master.ServerManager(1011): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 1, timeout of 4500 ms, interval of 1500 ms. 2015-06-17 11:08:12,696 INFO [M:0;10.0.0.110:51074] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,51074,1434564492618 with port=51074, startcode=1434564492618 2015-06-17 11:08:12,696 INFO [M:0;10.0.0.110:51074] master.ServerManager(459): Registering server=10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,696 INFO [M:0;10.0.0.110:51074] regionserver.HRegionServer(1341): Config from master: hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034 2015-06-17 11:08:12,697 INFO [M:0;10.0.0.110:51074] regionserver.HRegionServer(1341): Config from master: fs.defaultFS=hdfs://localhost:50895 2015-06-17 11:08:12,697 INFO [M:0;10.0.0.110:51074] regionserver.HRegionServer(1341): Config from master: hbase.master.info.port=-1 2015-06-17 11:08:12,697 WARN [M:0;10.0.0.110:51074] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:08:12,697 INFO [M:0;10.0.0.110:51074] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:08:12,697 DEBUG [M:0;10.0.0.110:51074] regionserver.HRegionServer(1603): logdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,700 DEBUG [M:0;10.0.0.110:51074] regionserver.Replication(142): ReplicationStatisticsThread 300 2015-06-17 11:08:12,700 INFO [M:0;10.0.0.110:51074] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:08:12,702 INFO [M:0;10.0.0.110:51074] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.0.0.110%2C51074%2C1434564492618.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51074,1434564492618, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:08:12,707 INFO [M:0;10.0.0.110:51074] wal.FSHLog(1365): Slow sync cost: 2 ms, current pipeline: [] 2015-06-17 11:08:12,707 INFO [M:0;10.0.0.110:51074] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51074,1434564492618/10.0.0.110%2C51074%2C1434564492618.default.1434564492702 2015-06-17 11:08:12,707 INFO [M:0;10.0.0.110:51074] regionserver.MetricsRegionServerWrapperImpl(113): Computing regionserver metrics every 5000 milliseconds 2015-06-17 11:08:12,708 DEBUG [M:0;10.0.0.110:51074] executor.ExecutorService(91): Starting executor service name=RS_OPEN_REGION-10.0.0.110:51074, corePoolSize=3, maxPoolSize=3 2015-06-17 11:08:12,708 DEBUG [M:0;10.0.0.110:51074] executor.ExecutorService(91): Starting executor service name=RS_OPEN_META-10.0.0.110:51074, corePoolSize=1, maxPoolSize=1 2015-06-17 11:08:12,708 DEBUG [M:0;10.0.0.110:51074] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_REGION-10.0.0.110:51074, corePoolSize=3, maxPoolSize=3 2015-06-17 11:08:12,708 DEBUG [M:0;10.0.0.110:51074] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_META-10.0.0.110:51074, corePoolSize=1, maxPoolSize=1 2015-06-17 11:08:12,709 DEBUG [M:0;10.0.0.110:51074] executor.ExecutorService(91): Starting executor service name=RS_LOG_REPLAY_OPS-10.0.0.110:51074, corePoolSize=2, maxPoolSize=2 2015-06-17 11:08:12,709 DEBUG [M:0;10.0.0.110:51074] executor.ExecutorService(91): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-10.0.0.110:51074, corePoolSize=3, maxPoolSize=3 2015-06-17 11:08:12,710 DEBUG [M:0;10.0.0.110:51074] zookeeper.ZKUtil(486): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:12,710 DEBUG [M:0;10.0.0.110:51074] zookeeper.ZKUtil(486): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,710 INFO [M:0;10.0.0.110:51074] regionserver.ReplicationSourceManager(229): Current list of replicators: [10.0.0.110,50905,1434564404764, 10.0.0.110,51074,1434564492618] other RSs: [10.0.0.110,50905,1434564404764, 10.0.0.110,51074,1434564492618] 2015-06-17 11:08:12,723 INFO [M:0;10.0.0.110:51074] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x4a20c70e connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:08:12,725 DEBUG [M:0;10.0.0.110:51074-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x4a20c70e0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:08:12,726 DEBUG [M:0;10.0.0.110:51074] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodecWithTags@319ae30e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:08:12,726 DEBUG [M:0;10.0.0.110:51074] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:08:12,726 DEBUG [M:0;10.0.0.110:51074] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:08:12,726 DEBUG [M:0;10.0.0.110:51074-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x4a20c70e-0x14e02b3a2320026 connected 2015-06-17 11:08:12,739 INFO [M:0;10.0.0.110:51074] regionserver.HeapMemoryManager(190): Starting HeapMemoryTuner chore. 2015-06-17 11:08:12,739 INFO [SplitLogWorker-10.0.0.110:51074] regionserver.SplitLogWorker(134): SplitLogWorker 10.0.0.110,51074,1434564492618 starting 2015-06-17 11:08:12,739 INFO [M:0;10.0.0.110:51074] regionserver.HRegionServer(1363): Serving as 10.0.0.110,51074,1434564492618, RpcServer on 10.0.0.110/10.0.0.110:51074, sessionid=0x14e02b3a2320022 2015-06-17 11:08:12,740 DEBUG [M:0;10.0.0.110:51074] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2015-06-17 11:08:12,740 DEBUG [M:0;10.0.0.110:51074] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager 10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,740 DEBUG [M:0;10.0.0.110:51074] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,51074,1434564492618' 2015-06-17 11:08:12,740 DEBUG [M:0;10.0.0.110:51074] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2015-06-17 11:08:12,741 DEBUG [M:0;10.0.0.110:51074] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2015-06-17 11:08:12,741 DEBUG [M:0;10.0.0.110:51074] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2015-06-17 11:08:12,741 DEBUG [M:0;10.0.0.110:51074] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2015-06-17 11:08:12,741 DEBUG [M:0;10.0.0.110:51074] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager 10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,741 DEBUG [M:0;10.0.0.110:51074] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,51074,1434564492618' 2015-06-17 11:08:12,741 DEBUG [M:0;10.0.0.110:51074] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2015-06-17 11:08:12,742 DEBUG [M:0;10.0.0.110:51074] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2015-06-17 11:08:12,742 DEBUG [M:0;10.0.0.110:51074] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2015-06-17 11:08:12,742 INFO [M:0;10.0.0.110:51074] quotas.RegionServerQuotaManager(62): Quota support disabled 2015-06-17 11:08:12,746 INFO [10.0.0.110:51074.activeMasterManager] master.ServerManager(1028): Finished waiting for region servers count to settle; checked in 1, slept for 50 ms, expecting minimum of 1, maximum of 1, master is running 2015-06-17 11:08:12,747 INFO [10.0.0.110:51074.activeMasterManager] master.ServerManager(459): Registering server=10.0.0.110,50905,1434564404764 2015-06-17 11:08:12,747 INFO [10.0.0.110:51074.activeMasterManager] master.HMaster(665): Registered server found up in zk but who has not yet reported in: 10.0.0.110,50905,1434564404764 2015-06-17 11:08:12,749 INFO [10.0.0.110:51074.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50905,1434564404764 belongs to an existing region server 2015-06-17 11:08:12,751 INFO [10.0.0.110:51074.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51074,1434564492618 belongs to an existing region server 2015-06-17 11:08:12,755 DEBUG [10.0.0.110:51074.activeMasterManager] coordination.ZKSplitLogManagerCoordination(650): Marked 1588230740 recovering from 10.0.0.110,51015,1434564463167: /hbase/recovering-regions/1588230740/10.0.0.110,51015,1434564463167 2015-06-17 11:08:12,755 INFO [10.0.0.110:51074.activeMasterManager] master.HMaster(884): Re-assigning hbase:meta with replicaId, 0 it was on 10.0.0.110,51015,1434564463167 2015-06-17 11:08:12,755 DEBUG [10.0.0.110:51074.activeMasterManager] master.AssignmentManager(1239): No previous transition plan found (or ignoring an existing plan) for hbase:meta,,1.1588230740; generated random plan=hri=hbase:meta,,1.1588230740, src=, dest=10.0.0.110,51074,1434564492618; 2 (online=2) available servers, forceNewPlan=false 2015-06-17 11:08:12,755 INFO [10.0.0.110:51074.activeMasterManager] master.AssignmentManager(1028): Assigning hbase:meta,,1.1588230740 to 10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,755 INFO [10.0.0.110:51074.activeMasterManager] master.RegionStates(1072): Transition {1588230740 state=OFFLINE, ts=1434564492755, server=10.0.0.110,51015,1434564463167} to {1588230740 state=PENDING_OPEN, ts=1434564492755, server=10.0.0.110,51074,1434564492618} 2015-06-17 11:08:12,755 INFO [10.0.0.110:51074.activeMasterManager] zookeeper.MetaTableLocator(440): Setting hbase:meta region location in ZooKeeper as 10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,756 DEBUG [10.0.0.110:51074.activeMasterManager] master.ServerManager(937): New admin connection to 10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,756 INFO [10.0.0.110:51074.activeMasterManager] regionserver.RSRpcServices(1432): Open hbase:meta,,1.1588230740 2015-06-17 11:08:12,757 DEBUG [10.0.0.110:51074.activeMasterManager] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564492756,"tag":[],"qualifier":"state","vlen":2}]},"row":"hbase:meta"} 2015-06-17 11:08:12,757 INFO [RS_OPEN_META-10.0.0.110:51074-0] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:08:12,759 INFO [RS_OPEN_META-10.0.0.110:51074-0] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.0.0.110%2C51074%2C1434564492618..meta, suffix=.meta, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51074,1434564492618, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:08:12,763 INFO [RS_OPEN_META-10.0.0.110:51074-0] wal.FSHLog(1365): Slow sync cost: 3 ms, current pipeline: [] 2015-06-17 11:08:12,763 INFO [RS_OPEN_META-10.0.0.110:51074-0] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51074,1434564492618/10.0.0.110%2C51074%2C1434564492618..meta.1434564492759.meta 2015-06-17 11:08:12,764 DEBUG [RS_OPEN_META-10.0.0.110:51074-0] regionserver.HRegion(6070): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2015-06-17 11:08:12,765 DEBUG [RS_OPEN_META-10.0.0.110:51074-0] coprocessor.CoprocessorHost(179): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2015-06-17 11:08:12,765 DEBUG [RS_OPEN_META-10.0.0.110:51074-0] regionserver.HRegion(7284): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2015-06-17 11:08:12,765 INFO [RS_OPEN_META-10.0.0.110:51074-0] regionserver.RegionCoprocessorHost(350): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2015-06-17 11:08:12,765 DEBUG [RS_OPEN_META-10.0.0.110:51074-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table meta 1588230740 2015-06-17 11:08:12,765 DEBUG [RS_OPEN_META-10.0.0.110:51074-0] regionserver.HRegion(722): Instantiated hbase:meta,,1.1588230740 2015-06-17 11:08:12,768 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:08:12,768 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:08:12,776 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/7acd1d3a04d44ab0a2c087c8d9edde78, isReference=false, isBulkLoadResult=false, seqid=40000117, majorCompaction=false 2015-06-17 11:08:12,782 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/96eb1546e8d44dd081fd3c2be70b7d3e, isReference=false, isBulkLoadResult=false, seqid=80000131, majorCompaction=false 2015-06-17 11:08:12,789 INFO [StoreFileOpenerThread-info-1] regionserver.StoreFile$Reader(1439): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fd7dd8c37dcd4f58adb7e614dd55c26b 2015-06-17 11:08:12,789 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/fd7dd8c37dcd4f58adb7e614dd55c26b, isReference=false, isBulkLoadResult=false, seqid=101, majorCompaction=false 2015-06-17 11:08:12,790 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:08:12,790 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:08:12,797 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/329b922a4c424a6b9d97b18b74f7f17e, isReference=false, isBulkLoadResult=false, seqid=40000117, majorCompaction=false 2015-06-17 11:08:12,802 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=81, retries=350, started=29234 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:12,803 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/adda056a1f284c47be8d8b15ee043a41, isReference=false, isBulkLoadResult=false, seqid=101, majorCompaction=false 2015-06-17 11:08:12,811 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/c394aa982f5044639d5cdde07e27d56e, isReference=false, isBulkLoadResult=false, seqid=80000131, majorCompaction=false 2015-06-17 11:08:12,813 DEBUG [RS_OPEN_META-10.0.0.110:51074-0] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740 2015-06-17 11:08:12,815 DEBUG [RS_OPEN_META-10.0.0.110:51074-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:08:12,818 DEBUG [RS_OPEN_META-10.0.0.110:51074-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/recovered.edits/120000134.seqid to file, newSeqId=120000134, maxSeqId=80000134 2015-06-17 11:08:12,818 INFO [RS_OPEN_META-10.0.0.110:51074-0] regionserver.HRegion(848): Onlined 1588230740; next sequenceid=120000134 2015-06-17 11:08:12,818 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1893): Post open deploy tasks for hbase:meta,,1.1588230740 2015-06-17 11:08:12,820 DEBUG [PostOpenDeployTasks:1588230740] regionserver.CompactSplitThread(356): Small Compaction requested: system; Because: Opening Region; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-06-17 11:08:12,820 DEBUG [PostOpenDeployTasks:1588230740] regionserver.CompactSplitThread(356): Small Compaction requested: system; Because: Opening Region; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-06-17 11:08:12,820 DEBUG [M:0;10.0.0.110:51074-shortCompactions-1434564492820] compactions.RatioBasedCompactionPolicy(92): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 10 blocking 2015-06-17 11:08:12,820 DEBUG [M:0;10.0.0.110:51074-longCompactions-1434564492637] compactions.RatioBasedCompactionPolicy(92): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 10 blocking 2015-06-17 11:08:12,822 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/recovering-regions/1588230740 2015-06-17 11:08:12,822 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(3181): Update last flushed sequence id of region 1588230740 for 10.0.0.110,51015,1434564463167 2015-06-17 11:08:12,823 DEBUG [PostOpenDeployTasks:1588230740] master.AssignmentManager(2812): Got transition OPENED for {1588230740 state=PENDING_OPEN, ts=1434564492755, server=10.0.0.110,51074,1434564492618} from 10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,823 INFO [PostOpenDeployTasks:1588230740] master.RegionStates(1072): Transition {1588230740 state=PENDING_OPEN, ts=1434564492755, server=10.0.0.110,51074,1434564492618} to {1588230740 state=OPEN, ts=1434564492823, server=10.0.0.110,51074,1434564492618} 2015-06-17 11:08:12,823 INFO [PostOpenDeployTasks:1588230740] zookeeper.MetaTableLocator(440): Setting hbase:meta region location in ZooKeeper as 10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,823 DEBUG [M:0;10.0.0.110:51074-shortCompactions-1434564492820] compactions.ExploringCompactionPolicy(122): Exploring compaction algorithm has selected 3 files of size 17171 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-06-17 11:08:12,823 DEBUG [M:0;10.0.0.110:51074-longCompactions-1434564492637] compactions.ExploringCompactionPolicy(122): Exploring compaction algorithm has selected 3 files of size 14490 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-06-17 11:08:12,823 DEBUG [PostOpenDeployTasks:1588230740] master.RegionStates(417): Onlined 1588230740 on 10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,824 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1920): Finished post open deploy task for hbase:meta,,1.1588230740 2015-06-17 11:08:12,824 DEBUG [RS_OPEN_META-10.0.0.110:51074-0] handler.OpenRegionHandler(125): Opened hbase:meta,,1.1588230740 on 10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,825 DEBUG [M:0;10.0.0.110:51074-shortCompactions-1434564492820] regionserver.HStore(1568): 1588230740 - info: Initiating minor compaction (all files) 2015-06-17 11:08:12,825 DEBUG [M:0;10.0.0.110:51074-longCompactions-1434564492637] regionserver.HStore(1568): 1588230740 - table: Initiating minor compaction (all files) 2015-06-17 11:08:12,825 INFO [M:0;10.0.0.110:51074-shortCompactions-1434564492820] regionserver.HRegion(1784): Starting compaction on info in region hbase:meta,,1.1588230740 2015-06-17 11:08:12,825 INFO [M:0;10.0.0.110:51074-longCompactions-1434564492637] regionserver.HRegion(1784): Starting compaction on table in region hbase:meta,,1.1588230740 2015-06-17 11:08:12,825 INFO [M:0;10.0.0.110:51074-longCompactions-1434564492637] regionserver.HStore(1196): Starting compaction of 3 file(s) in table of hbase:meta,,1.1588230740 into tmpdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp, totalSize=14.2 K 2015-06-17 11:08:12,825 INFO [M:0;10.0.0.110:51074-shortCompactions-1434564492820] regionserver.HStore(1196): Starting compaction of 3 file(s) in info of hbase:meta,,1.1588230740 into tmpdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp, totalSize=16.8 K 2015-06-17 11:08:12,826 DEBUG [M:0;10.0.0.110:51074-longCompactions-1434564492637] compactions.Compactor(169): Compacting hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/adda056a1f284c47be8d8b15ee043a41, keycount=4, bloomtype=NONE, size=4.8 K, encoding=NONE, seqNum=101, earliestPutTs=1434564405380 2015-06-17 11:08:12,826 DEBUG [M:0;10.0.0.110:51074-shortCompactions-1434564492820] compactions.Compactor(169): Compacting hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/fd7dd8c37dcd4f58adb7e614dd55c26b, keycount=11, bloomtype=NONE, size=6.1 K, encoding=NONE, seqNum=101, earliestPutTs=1434564406293 2015-06-17 11:08:12,827 DEBUG [M:0;10.0.0.110:51074-longCompactions-1434564492637] compactions.Compactor(169): Compacting hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/329b922a4c424a6b9d97b18b74f7f17e, keycount=3, bloomtype=NONE, size=4.7 K, encoding=NONE, seqNum=40000117, earliestPutTs=1434564430159 2015-06-17 11:08:12,827 DEBUG [M:0;10.0.0.110:51074-shortCompactions-1434564492820] compactions.Compactor(169): Compacting hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/7acd1d3a04d44ab0a2c087c8d9edde78, keycount=6, bloomtype=NONE, size=5.3 K, encoding=NONE, seqNum=40000117, earliestPutTs=1434564430428 2015-06-17 11:08:12,827 DEBUG [M:0;10.0.0.110:51074-longCompactions-1434564492637] compactions.Compactor(169): Compacting hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/c394aa982f5044639d5cdde07e27d56e, keycount=2, bloomtype=NONE, size=4.6 K, encoding=NONE, seqNum=80000131, earliestPutTs=1434564463304 2015-06-17 11:08:12,827 DEBUG [M:0;10.0.0.110:51074-shortCompactions-1434564492820] compactions.Compactor(169): Compacting hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/96eb1546e8d44dd081fd3c2be70b7d3e, keycount=6, bloomtype=NONE, size=5.3 K, encoding=NONE, seqNum=80000131, earliestPutTs=1434564463549 2015-06-17 11:08:12,834 INFO [M:0;10.0.0.110:51074-shortCompactions-1434564492820] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:08:12,834 INFO [M:0;10.0.0.110:51074-longCompactions-1434564492637] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:08:12,839 INFO [M:0;10.0.0.110:51074-longCompactions-1434564492637] compactions.PressureAwareCompactionThroughputController(239): hbase:meta,,1.1588230740#table average throughput is 0.35 MB/sec, slept 0 time(s) and total slept time is 0 ms. 1 active compactions remaining, total limit is 10.00 MB/sec 2015-06-17 11:08:12,839 INFO [M:0;10.0.0.110:51074-shortCompactions-1434564492820] compactions.PressureAwareCompactionThroughputController(239): hbase:meta,,1.1588230740#info average throughput is 1.71 MB/sec, slept 0 time(s) and total slept time is 0 ms. 0 active compactions remaining, total limit is 10.00 MB/sec 2015-06-17 11:08:12,844 INFO [IPC Server handler 6 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741875_1051{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 5079 2015-06-17 11:08:12,845 INFO [IPC Server handler 5 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741876_1052{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 6628 2015-06-17 11:08:12,854 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=241, retries=350, started=62406 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:12,897 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:08:12,899 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:12,899 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51074: callId: 1187 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:12,902 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:12,902 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51074: callId: 1189 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:12,906 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:12,906 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51074: callId: 1191 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:12,912 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:12,912 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51074: callId: 1193 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:12,924 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:12,924 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51074: callId: 1195 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:12,945 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:12,945 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51074: callId: 1197 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:12,959 DEBUG [AsyncRpcChannel-pool2-t8] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service ClientService, sasl=false 2015-06-17 11:08:12,959 DEBUG [RpcServer.listener,port=51074] ipc.RpcServer$Listener(840): RpcServer.listener,port=51074: connection from 10.0.0.110:51094; # active connections: 2 2015-06-17 11:08:12,960 INFO [RpcServer.reader=2,bindAddress=10.0.0.110,port=51074] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51094 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:08:12,962 INFO [10.0.0.110:51074.activeMasterManager] hbase.MetaTableAccessor(1692): Updated table hbase:meta state to ENABLED in META 2015-06-17 11:08:12,963 DEBUG [10.0.0.110:51074.activeMasterManager] master.MasterFileSystem(339): Renamed region directory: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51015,1434564463167-splitting 2015-06-17 11:08:12,963 INFO [10.0.0.110:51074.activeMasterManager] master.SplitLogManager(560): dead splitlog workers [10.0.0.110,51015,1434564463167] 2015-06-17 11:08:12,964 INFO [10.0.0.110:51074.activeMasterManager] master.SplitLogManager(192): hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51015,1434564463167-splitting is empty dir, no logs to split 2015-06-17 11:08:12,965 DEBUG [10.0.0.110:51074.activeMasterManager] master.SplitLogManager(255): Scheduling batch of logs to split 2015-06-17 11:08:12,965 INFO [10.0.0.110:51074.activeMasterManager] master.SplitLogManager(257): started splitting 0 logs in [hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51015,1434564463167-splitting] for [10.0.0.110,51015,1434564463167] 2015-06-17 11:08:12,965 DEBUG [10.0.0.110:51074.activeMasterManager] coordination.ZKSplitLogManagerCoordination(324): Processing recovering [1588230740] and servers [10.0.0.110,51015,1434564463167], isMetaRecovery=true 2015-06-17 11:08:12,967 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/recovering-regions/1588230740/10.0.0.110,51015,1434564463167 2015-06-17 11:08:12,968 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/recovering-regions/1588230740 2015-06-17 11:08:12,969 INFO [RS_OPEN_REGION-10.0.0.110:51074-0] handler.FinishRegionRecoveringHandler(52): /hbase/recovering-regions/1588230740 deleted; 1588230740 recovered. 2015-06-17 11:08:12,969 INFO [10.0.0.110:51074.activeMasterManager] master.SplitLogManager(316): finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51015,1434564463167-splitting] in 4ms 2015-06-17 11:08:12,970 DEBUG [10.0.0.110:51074.activeMasterManager] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564492969,"tag":[],"qualifier":"state","vlen":2}]},"row":"hbase:meta"} 2015-06-17 11:08:12,972 INFO [10.0.0.110:51074.activeMasterManager] hbase.MetaTableAccessor(1692): Updated table hbase:meta state to ENABLED in META 2015-06-17 11:08:12,978 INFO [10.0.0.110:51074.activeMasterManager] master.ServerManager(686): AssignmentManager hasn't finished failover cleanup; waiting 2015-06-17 11:08:12,979 INFO [10.0.0.110:51074.activeMasterManager] master.HMaster(911): hbase:meta with replicaId 0 assigned=1, location=10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,987 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:12,987 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51074: callId: 1199 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:12,997 INFO [10.0.0.110:51074.activeMasterManager] master.RegionStates(1072): Transition {9ccc7b1e8942a5472c9cf2d844c2fff0 state=OPEN, ts=1434564492993, server=10.0.0.110,51015,1434564463167} to {9ccc7b1e8942a5472c9cf2d844c2fff0 state=CLOSED, ts=1434564492997, server=10.0.0.110,51015,1434564463167} 2015-06-17 11:08:12,997 INFO [10.0.0.110:51074.activeMasterManager] master.AssignmentManager(538): Clean cluster startup. Assigning user regions 2015-06-17 11:08:12,997 INFO [10.0.0.110:51074.activeMasterManager] master.AssignmentManager(680): Assigning 1 region(s) to 10.0.0.110,51074,1434564492618 2015-06-17 11:08:12,999 INFO [10.0.0.110:51074.activeMasterManager] master.RegionStates(1072): Transition {9ccc7b1e8942a5472c9cf2d844c2fff0 state=CLOSED, ts=1434564492997, server=10.0.0.110,51015,1434564463167} to {9ccc7b1e8942a5472c9cf2d844c2fff0 state=PENDING_OPEN, ts=1434564492999, server=10.0.0.110,51074,1434564492618} 2015-06-17 11:08:12,999 INFO [10.0.0.110:51074.activeMasterManager] master.RegionStateStore(207): Updating row hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. with state=PENDING_OPEN&sn=10.0.0.110,51074,1434564492618 2015-06-17 11:08:13,000 INFO [10.0.0.110:51074.activeMasterManager] regionserver.RSRpcServices(1432): Open hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:08:13,001 DEBUG [10.0.0.110:51074.activeMasterManager] master.AssignmentManager(851): Bulk assigning done for 10.0.0.110,51074,1434564492618 2015-06-17 11:08:13,001 DEBUG [RS_OPEN_REGION-10.0.0.110:51074-1] regionserver.HRegion(6070): Opening region: {ENCODED => 9ccc7b1e8942a5472c9cf2d844c2fff0, NAME => 'hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.', STARTKEY => '', ENDKEY => ''} 2015-06-17 11:08:13,002 DEBUG [RS_OPEN_REGION-10.0.0.110:51074-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table namespace 9ccc7b1e8942a5472c9cf2d844c2fff0 2015-06-17 11:08:13,002 DEBUG [RS_OPEN_REGION-10.0.0.110:51074-1] regionserver.HRegion(722): Instantiated hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:08:13,003 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=82, retries=350, started=29435 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:13,004 INFO [StoreOpener-9ccc7b1e8942a5472c9cf2d844c2fff0-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=6, currentSize=1078728, freeSize=1042883576, maxSize=1043962304, heapSize=1078728, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:08:13,004 INFO [StoreOpener-9ccc7b1e8942a5472c9cf2d844c2fff0-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:08:13,010 INFO [10.0.0.110:51074.activeMasterManager] master.AssignmentManager(420): Joined the cluster in 30ms, failover=false 2015-06-17 11:08:13,013 DEBUG [StoreOpener-9ccc7b1e8942a5472c9cf2d844c2fff0-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/info/b88c94ff75ab4fafba9b040819600c6e, isReference=false, isBulkLoadResult=false, seqid=8, majorCompaction=false 2015-06-17 11:08:13,014 DEBUG [10.0.0.110:51074.activeMasterManager] zookeeper.ZKUtil(486): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2015-06-17 11:08:13,015 DEBUG [RS_OPEN_REGION-10.0.0.110:51074-1] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0 2015-06-17 11:08:13,015 DEBUG [10.0.0.110:51074.activeMasterManager] hbase.ZKNamespaceManager(196): Updating namespace cache from node default with data: \x0A\x07default 2015-06-17 11:08:13,015 DEBUG [10.0.0.110:51074.activeMasterManager] hbase.ZKNamespaceManager(196): Updating namespace cache from node hbase with data: \x0A\x05hbase 2015-06-17 11:08:13,017 DEBUG [RS_OPEN_REGION-10.0.0.110:51074-1] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:08:13,020 DEBUG [RS_OPEN_REGION-10.0.0.110:51074-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/recovered.edits/18.seqid to file, newSeqId=18, maxSeqId=17 2015-06-17 11:08:13,021 INFO [RS_OPEN_REGION-10.0.0.110:51074-1] regionserver.HRegion(848): Onlined 9ccc7b1e8942a5472c9cf2d844c2fff0; next sequenceid=18 2015-06-17 11:08:13,022 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] regionserver.HRegionServer(1893): Post open deploy tasks for hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:08:13,022 DEBUG [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.AssignmentManager(2812): Got transition OPENED for {9ccc7b1e8942a5472c9cf2d844c2fff0 state=PENDING_OPEN, ts=1434564492999, server=10.0.0.110,51074,1434564492618} from 10.0.0.110,51074,1434564492618 2015-06-17 11:08:13,022 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStates(1072): Transition {9ccc7b1e8942a5472c9cf2d844c2fff0 state=PENDING_OPEN, ts=1434564492999, server=10.0.0.110,51074,1434564492618} to {9ccc7b1e8942a5472c9cf2d844c2fff0 state=OPEN, ts=1434564493022, server=10.0.0.110,51074,1434564492618} 2015-06-17 11:08:13,022 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStateStore(207): Updating row hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. with state=OPEN&openSeqNum=18&server=10.0.0.110,51074,1434564492618 2015-06-17 11:08:13,023 DEBUG [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStates(417): Onlined 9ccc7b1e8942a5472c9cf2d844c2fff0 on 10.0.0.110,51074,1434564492618 2015-06-17 11:08:13,023 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStates(429): Offlined 9ccc7b1e8942a5472c9cf2d844c2fff0 from 10.0.0.110,51015,1434564463167 2015-06-17 11:08:13,025 DEBUG [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] regionserver.HRegionServer(1920): Finished post open deploy task for hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:08:13,025 DEBUG [RS_OPEN_REGION-10.0.0.110:51074-1] handler.OpenRegionHandler(125): Opened hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. on 10.0.0.110,51074,1434564492618 2015-06-17 11:08:13,055 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=242, retries=350, started=62607 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:13,088 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:13,088 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51074: callId: 1201 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:13,191 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:13,191 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51074: callId: 1203 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:13,204 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=83, retries=350, started=29636 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:13,254 DEBUG [M:0;10.0.0.110:51074-shortCompactions-1434564492820] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/b97cb9bad56b4097bc8516a91cd2d5c4 as hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/b97cb9bad56b4097bc8516a91cd2d5c4 2015-06-17 11:08:13,254 DEBUG [M:0;10.0.0.110:51074-longCompactions-1434564492637] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/309e63f83f3a4850bf003589102a7393 as hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/309e63f83f3a4850bf003589102a7393 2015-06-17 11:08:13,257 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=243, retries=350, started=62809 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:13,266 DEBUG [M:0;10.0.0.110:51074-longCompactions-1434564492637] regionserver.HStore(1703): Removing store files after compaction... 2015-06-17 11:08:13,266 DEBUG [M:0;10.0.0.110:51074-shortCompactions-1434564492820] regionserver.HStore(1703): Removing store files after compaction... 2015-06-17 11:08:13,268 DEBUG [M:0;10.0.0.110:51074-longCompactions-1434564492637] backup.HFileArchiver(224): Archiving compacted store files. 2015-06-17 11:08:13,268 DEBUG [M:0;10.0.0.110:51074-shortCompactions-1434564492820] backup.HFileArchiver(224): Archiving compacted store files. 2015-06-17 11:08:13,272 DEBUG [M:0;10.0.0.110:51074-shortCompactions-1434564492820] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/fd7dd8c37dcd4f58adb7e614dd55c26b, to hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/archive/data/hbase/meta/1588230740/info/fd7dd8c37dcd4f58adb7e614dd55c26b 2015-06-17 11:08:13,272 DEBUG [M:0;10.0.0.110:51074-longCompactions-1434564492637] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/adda056a1f284c47be8d8b15ee043a41, to hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/archive/data/hbase/meta/1588230740/table/adda056a1f284c47be8d8b15ee043a41 2015-06-17 11:08:13,273 DEBUG [M:0;10.0.0.110:51074-shortCompactions-1434564492820] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/7acd1d3a04d44ab0a2c087c8d9edde78, to hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/archive/data/hbase/meta/1588230740/info/7acd1d3a04d44ab0a2c087c8d9edde78 2015-06-17 11:08:13,274 DEBUG [M:0;10.0.0.110:51074-longCompactions-1434564492637] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/329b922a4c424a6b9d97b18b74f7f17e, to hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/archive/data/hbase/meta/1588230740/table/329b922a4c424a6b9d97b18b74f7f17e 2015-06-17 11:08:13,276 DEBUG [M:0;10.0.0.110:51074-shortCompactions-1434564492820] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/96eb1546e8d44dd081fd3c2be70b7d3e, to hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/archive/data/hbase/meta/1588230740/info/96eb1546e8d44dd081fd3c2be70b7d3e 2015-06-17 11:08:13,277 DEBUG [M:0;10.0.0.110:51074-longCompactions-1434564492637] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/c394aa982f5044639d5cdde07e27d56e, to hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/archive/data/hbase/meta/1588230740/table/c394aa982f5044639d5cdde07e27d56e 2015-06-17 11:08:13,277 INFO [M:0;10.0.0.110:51074-shortCompactions-1434564492820] regionserver.HStore(1324): Completed compaction of 3 (all) file(s) in info of hbase:meta,,1.1588230740 into b97cb9bad56b4097bc8516a91cd2d5c4(size=6.5 K), total size for store is 6.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2015-06-17 11:08:13,277 INFO [M:0;10.0.0.110:51074-longCompactions-1434564492637] regionserver.HStore(1324): Completed compaction of 3 (all) file(s) in table of hbase:meta,,1.1588230740 into 309e63f83f3a4850bf003589102a7393(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2015-06-17 11:08:13,278 INFO [M:0;10.0.0.110:51074-longCompactions-1434564492637] regionserver.CompactSplitThread$CompactionRunner(527): Completed compaction: Request = regionName=hbase:meta,,1.1588230740, storeName=table, fileCount=3, fileSize=14.2 K, priority=7, time=1434564492825012000; duration=0sec 2015-06-17 11:08:13,278 INFO [M:0;10.0.0.110:51074-shortCompactions-1434564492820] regionserver.CompactSplitThread$CompactionRunner(527): Completed compaction: Request = regionName=hbase:meta,,1.1588230740, storeName=info, fileCount=3, fileSize=16.8 K, priority=7, time=1434564492825012000; duration=0sec 2015-06-17 11:08:13,278 DEBUG [M:0;10.0.0.110:51074-longCompactions-1434564492637] regionserver.CompactSplitThread$CompactionRunner(550): CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-06-17 11:08:13,278 DEBUG [M:0;10.0.0.110:51074-shortCompactions-1434564492820] regionserver.CompactSplitThread$CompactionRunner(550): CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-06-17 11:08:13,293 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:13,293 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51074: callId: 1205 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:13,395 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:13,395 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51074: callId: 1207 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:13,395 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=498 ms ago, cancelled=false, msg= 2015-06-17 11:08:13,405 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=84, retries=350, started=29836 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:13,457 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=244, retries=350, started=63009 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:13,598 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:13,598 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51074: callId: 1209 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:13,598 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=701 ms ago, cancelled=false, msg= 2015-06-17 11:08:13,606 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=85, retries=350, started=30038 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:13,657 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=245, retries=350, started=63209 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:13,799 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:13,799 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51074: callId: 1211 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:13,800 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=903 ms ago, cancelled=false, msg= 2015-06-17 11:08:13,808 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=86, retries=350, started=30240 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:13,857 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=246, retries=350, started=63409 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:14,001 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:14,001 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51074: callId: 1213 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:14,002 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1105 ms ago, cancelled=false, msg= 2015-06-17 11:08:14,008 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=87, retries=350, started=30440 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:14,058 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=247, retries=350, started=63610 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:14,205 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:14,205 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51074: callId: 1215 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:14,205 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1308 ms ago, cancelled=false, msg= 2015-06-17 11:08:14,209 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=88, retries=350, started=30641 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:14,259 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=248, retries=350, started=63811 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:14,407 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:14,407 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51074: callId: 1217 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:14,407 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1510 ms ago, cancelled=false, msg= 2015-06-17 11:08:14,410 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=89, retries=350, started=30842 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:14,460 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=249, retries=350, started=64012 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:14,487 DEBUG [AsyncRpcChannel-pool2-t10] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2015-06-17 11:08:14,488 DEBUG [RpcServer.listener,port=51074] ipc.RpcServer$Listener(840): RpcServer.listener,port=51074: connection from 10.0.0.110:51107; # active connections: 3 2015-06-17 11:08:14,490 INFO [RpcServer.reader=3,bindAddress=10.0.0.110,port=51074] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51107 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:08:14,609 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:14,610 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51074: callId: 1219 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:14,610 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1713 ms ago, cancelled=false, msg= 2015-06-17 11:08:14,611 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=90, retries=350, started=31043 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:14,661 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=250, retries=350, started=64213 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:14,811 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=91, retries=350, started=31243 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:14,812 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:14,812 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51074: callId: 1221 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:14,813 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1916 ms ago, cancelled=false, msg= 2015-06-17 11:08:14,863 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=251, retries=350, started=64415 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:15,013 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=92, retries=350, started=31445 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:15,014 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:15,014 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51074: callId: 1223 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:15,015 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2118 ms ago, cancelled=false, msg= 2015-06-17 11:08:15,063 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=252, retries=350, started=64615 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:15,215 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=93, retries=350, started=31647 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:15,216 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:15,216 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51074: callId: 1225 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:15,217 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2320 ms ago, cancelled=false, msg= 2015-06-17 11:08:15,265 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=253, retries=350, started=64817 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:15,417 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=94, retries=350, started=31849 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:15,419 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:15,420 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51074: callId: 1227 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:15,420 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2523 ms ago, cancelled=false, msg= 2015-06-17 11:08:15,465 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=254, retries=350, started=65017 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:15,618 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=95, retries=350, started=32050 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:15,623 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:15,623 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51074: callId: 1229 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:15,623 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2726 ms ago, cancelled=false, msg= 2015-06-17 11:08:15,666 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=255, retries=350, started=65218 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:15,819 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=96, retries=350, started=32251 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:15,824 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:15,825 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51074: callId: 1231 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:15,825 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2928 ms ago, cancelled=false, msg= 2015-06-17 11:08:15,866 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=256, retries=350, started=65418 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:16,020 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=97, retries=350, started=32452 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:16,027 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:16,027 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51074: callId: 1233 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:16,028 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3131 ms ago, cancelled=false, msg= 2015-06-17 11:08:16,067 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=257, retries=350, started=65619 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:16,223 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=98, retries=350, started=32655 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:16,230 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:16,230 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51074: callId: 1235 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:16,231 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3334 ms ago, cancelled=false, msg= 2015-06-17 11:08:16,268 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=258, retries=350, started=65820 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:16,423 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=99, retries=350, started=32855 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:16,432 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:16,432 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51074: callId: 1237 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:16,432 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3535 ms ago, cancelled=false, msg= 2015-06-17 11:08:16,469 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=259, retries=350, started=66021 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:16,625 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=100, retries=350, started=33057 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:16,634 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:16,634 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51074: callId: 1239 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:16,635 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3738 ms ago, cancelled=false, msg= 2015-06-17 11:08:16,670 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=260, retries=350, started=66222 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:16,826 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=101, retries=350, started=33258 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:16,836 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:16,836 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51074: callId: 1241 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:16,836 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3939 ms ago, cancelled=false, msg= 2015-06-17 11:08:16,871 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=261, retries=350, started=66423 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:17,027 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=102, retries=350, started=33458 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:17,037 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:17,037 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51074: callId: 1243 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:17,038 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4141 ms ago, cancelled=false, msg= 2015-06-17 11:08:17,072 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=262, retries=350, started=66624 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:17,227 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=103, retries=350, started=33659 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:17,240 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:17,240 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51074: callId: 1245 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:17,240 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4343 ms ago, cancelled=false, msg= 2015-06-17 11:08:17,274 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=263, retries=350, started=66826 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:17,429 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=104, retries=350, started=33861 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:17,441 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:17,441 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51074: callId: 1247 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:17,442 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4545 ms ago, cancelled=false, msg= 2015-06-17 11:08:17,474 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=264, retries=350, started=67026 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:17,630 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=105, retries=350, started=34062 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:17,643 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:17,643 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51074: callId: 1249 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:17,643 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4746 ms ago, cancelled=false, msg= 2015-06-17 11:08:17,676 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=265, retries=350, started=67228 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:17,831 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=106, retries=350, started=34263 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:17,846 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:17,846 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51074: callId: 1251 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:17,846 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4949 ms ago, cancelled=false, msg= 2015-06-17 11:08:17,876 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=266, retries=350, started=67428 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:18,032 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=107, retries=350, started=34464 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:18,048 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:18,049 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51074: callId: 1253 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:18,049 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5152 ms ago, cancelled=false, msg= 2015-06-17 11:08:18,077 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=267, retries=350, started=67629 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:18,234 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=108, retries=350, started=34666 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:18,250 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:18,250 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51074: callId: 1255 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:18,250 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5353 ms ago, cancelled=false, msg= 2015-06-17 11:08:18,251 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564492897, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:08:18,277 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=268, retries=350, started=67829 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:18,436 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=109, retries=350, started=34868 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:18,453 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:18,453 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51074: callId: 1257 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:18,456 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:18,457 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51074: callId: 1259 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:18,461 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:18,461 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51074: callId: 1261 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:18,467 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:18,467 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51074: callId: 1263 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:18,478 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=269, retries=350, started=68030 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:18,479 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:18,479 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51074: callId: 1265 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:18,500 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:18,500 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51074: callId: 1267 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:18,542 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:18,542 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51074: callId: 1269 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:18,637 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=110, retries=350, started=35069 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:18,644 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:18,644 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51074: callId: 1271 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:18,679 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=270, retries=350, started=68231 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:18,745 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:18,745 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51074: callId: 1273 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:18,838 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=111, retries=350, started=35270 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:18,847 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:18,847 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51074: callId: 1275 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:18,880 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=271, retries=350, started=68432 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:18,948 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:18,948 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51074: callId: 1277 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:18,948 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=496 ms ago, cancelled=false, msg= 2015-06-17 11:08:19,038 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=112, retries=350, started=35470 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:19,081 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=272, retries=350, started=68633 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:19,151 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:19,152 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51074: callId: 1279 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:19,152 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=700 ms ago, cancelled=false, msg= 2015-06-17 11:08:19,239 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=113, retries=350, started=35671 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:19,282 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=273, retries=350, started=68834 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:19,355 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:19,355 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51074: callId: 1281 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:19,356 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=904 ms ago, cancelled=false, msg= 2015-06-17 11:08:19,439 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=114, retries=350, started=35871 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:19,484 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=274, retries=350, started=69036 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:19,557 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:19,557 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51074: callId: 1283 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:19,558 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1106 ms ago, cancelled=false, msg= 2015-06-17 11:08:19,640 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=115, retries=350, started=36072 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:19,686 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=275, retries=350, started=69238 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:19,759 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:19,759 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51074: callId: 1285 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:19,760 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1308 ms ago, cancelled=false, msg= 2015-06-17 11:08:19,841 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=116, retries=350, started=36273 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:19,887 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=276, retries=350, started=69439 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:19,962 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:19,962 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51074: callId: 1287 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:19,962 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1510 ms ago, cancelled=false, msg= 2015-06-17 11:08:20,042 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=117, retries=350, started=36474 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:20,089 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=277, retries=350, started=69641 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:20,164 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:20,165 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51074: callId: 1289 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:20,165 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1713 ms ago, cancelled=false, msg= 2015-06-17 11:08:20,242 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=118, retries=350, started=36674 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:20,290 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=278, retries=350, started=69842 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:20,366 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:20,367 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51074: callId: 1291 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:20,367 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1915 ms ago, cancelled=false, msg= 2015-06-17 11:08:20,444 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=119, retries=350, started=36876 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:20,492 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=279, retries=350, started=70043 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:20,570 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:20,571 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51074: callId: 1293 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:20,571 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2119 ms ago, cancelled=false, msg= 2015-06-17 11:08:20,644 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=120, retries=350, started=37076 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:20,693 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=280, retries=350, started=70245 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:20,774 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:20,774 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51074: callId: 1295 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:20,774 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2322 ms ago, cancelled=false, msg= 2015-06-17 11:08:20,846 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=121, retries=350, started=37278 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:20,893 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=281, retries=350, started=70445 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:20,977 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:20,977 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51074: callId: 1297 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:20,978 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2526 ms ago, cancelled=false, msg= 2015-06-17 11:08:21,046 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=122, retries=350, started=37478 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:21,094 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=282, retries=350, started=70646 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:21,181 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:21,181 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51074: callId: 1299 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:21,181 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2729 ms ago, cancelled=false, msg= 2015-06-17 11:08:21,246 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=123, retries=350, started=37678 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:21,296 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=283, retries=350, started=70848 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:21,383 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:21,383 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51074: callId: 1301 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:21,383 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2931 ms ago, cancelled=false, msg= 2015-06-17 11:08:21,447 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=124, retries=350, started=37879 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:21,497 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=284, retries=350, started=71049 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:21,584 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:21,584 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51074: callId: 1303 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:21,585 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3133 ms ago, cancelled=false, msg= 2015-06-17 11:08:21,648 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=125, retries=350, started=38080 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:21,700 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=285, retries=350, started=71252 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:21,787 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:21,787 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51074: callId: 1305 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:21,787 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3335 ms ago, cancelled=false, msg= 2015-06-17 11:08:21,850 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=126, retries=350, started=38282 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:21,900 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=286, retries=350, started=71452 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:21,990 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:21,990 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51074: callId: 1307 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:21,990 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3538 ms ago, cancelled=false, msg= 2015-06-17 11:08:22,050 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=127, retries=350, started=38482 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:22,101 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=287, retries=350, started=71653 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:22,191 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:22,191 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51074: callId: 1309 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:22,192 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3740 ms ago, cancelled=false, msg= 2015-06-17 11:08:22,250 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=128, retries=350, started=38682 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:22,302 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=288, retries=350, started=71854 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:22,394 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:22,394 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51074: callId: 1311 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:22,394 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3942 ms ago, cancelled=false, msg= 2015-06-17 11:08:22,451 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=129, retries=350, started=38883 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:22,504 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=289, retries=350, started=72056 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:22,595 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:22,596 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51074: callId: 1313 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:22,596 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4144 ms ago, cancelled=false, msg= 2015-06-17 11:08:22,652 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=130, retries=350, started=39084 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:22,665 DEBUG [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(188): Sleeping for 3500 before executing next master restart count=6 2015-06-17 11:08:22,704 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=290, retries=350, started=72256 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:22,797 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:22,797 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51074: callId: 1315 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:22,798 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4346 ms ago, cancelled=false, msg= 2015-06-17 11:08:22,853 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=131, retries=350, started=39285 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:22,904 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=291, retries=350, started=72456 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:23,000 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:23,000 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51074: callId: 1317 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:23,000 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4548 ms ago, cancelled=false, msg= 2015-06-17 11:08:23,054 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=132, retries=350, started=39486 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:23,106 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=292, retries=350, started=72658 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:23,202 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:23,202 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51074: callId: 1319 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:23,202 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4750 ms ago, cancelled=false, msg= 2015-06-17 11:08:23,255 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=133, retries=350, started=39687 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:23,308 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=293, retries=350, started=72860 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:23,403 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:23,404 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51074: callId: 1321 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:23,404 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4952 ms ago, cancelled=false, msg= 2015-06-17 11:08:23,456 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=134, retries=350, started=39888 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:23,509 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=294, retries=350, started=73061 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:23,607 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:23,607 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51074: callId: 1323 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:23,607 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5155 ms ago, cancelled=false, msg= 2015-06-17 11:08:23,658 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=135, retries=350, started=40090 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:23,711 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=295, retries=350, started=73263 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:23,809 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:23,809 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51074: callId: 1325 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:23,809 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5357 ms ago, cancelled=false, msg= 2015-06-17 11:08:23,810 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564498452, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:08:23,858 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=136, retries=350, started=40290 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:23,911 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=296, retries=350, started=73463 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:24,012 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:24,012 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51074: callId: 1327 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:24,015 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:24,015 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51074: callId: 1329 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:24,019 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:24,019 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51074: callId: 1331 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:24,025 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:24,026 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51074: callId: 1333 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:24,038 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:24,038 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51074: callId: 1335 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:24,059 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=137, retries=350, started=40491 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:24,060 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:24,060 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51074: callId: 1337 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:24,102 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:24,102 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51074: callId: 1339 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:24,114 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=297, retries=350, started=73666 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:24,204 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:24,204 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51074: callId: 1341 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:24,259 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=138, retries=350, started=40691 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:24,306 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:24,306 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51074: callId: 1343 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:24,316 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=298, retries=350, started=73868 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:24,407 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:24,407 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51074: callId: 1345 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:24,459 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=139, retries=350, started=40891 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:24,508 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:24,508 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51074: callId: 1347 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:24,509 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=498 ms ago, cancelled=false, msg= 2015-06-17 11:08:24,516 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=299, retries=350, started=74068 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:24,661 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=140, retries=350, started=41093 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:24,711 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:24,711 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51074: callId: 1349 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:24,711 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=700 ms ago, cancelled=false, msg= 2015-06-17 11:08:24,717 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=300, retries=350, started=74269 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:24,862 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=141, retries=350, started=41294 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:24,913 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:24,913 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51074: callId: 1351 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:24,914 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=903 ms ago, cancelled=false, msg= 2015-06-17 11:08:24,917 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=301, retries=350, started=74469 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:25,062 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=142, retries=350, started=41494 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:25,115 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:25,115 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51074: callId: 1353 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:25,115 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1104 ms ago, cancelled=false, msg= 2015-06-17 11:08:25,118 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=302, retries=350, started=74670 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:25,263 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=143, retries=350, started=41695 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:25,319 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:25,319 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=303, retries=350, started=74871 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:25,319 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51074: callId: 1355 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:25,320 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1309 ms ago, cancelled=false, msg= 2015-06-17 11:08:25,464 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=144, retries=350, started=41896 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:25,521 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=304, retries=350, started=75073 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:25,523 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:25,523 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51074: callId: 1357 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:25,523 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1512 ms ago, cancelled=false, msg= 2015-06-17 11:08:25,666 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=145, retries=350, started=42098 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:25,723 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=305, retries=350, started=75275 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:25,725 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:25,725 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51074: callId: 1359 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:25,726 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1715 ms ago, cancelled=false, msg= 2015-06-17 11:08:25,868 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=146, retries=350, started=42300 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:25,924 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=306, retries=350, started=75476 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:25,927 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:25,927 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51074: callId: 1361 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:25,927 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1916 ms ago, cancelled=false, msg= 2015-06-17 11:08:26,070 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=147, retries=350, started=42502 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:26,125 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=307, retries=350, started=75677 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:26,130 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:26,130 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51074: callId: 1363 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:26,130 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2119 ms ago, cancelled=false, msg= 2015-06-17 11:08:26,168 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(197): executing master kill 2015-06-17 11:08:26,168 INFO [MasterRestartThread] hbase.MiniHBaseCluster(422): Aborting 10.0.0.110,51074,1434564492618 2015-06-17 11:08:26,168 FATAL [MasterRestartThread] master.HMaster(1953): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2015-06-17 11:08:26,168 FATAL [MasterRestartThread] master.HMaster(1956): Aborting for tests java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:423) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:270) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$5.run(TestMasterRestartWithProcedures.java:198) 2015-06-17 11:08:26,168 INFO [MasterRestartThread] regionserver.HRegionServer(1859): STOPPED: Aborting for tests 2015-06-17 11:08:26,168 INFO [MasterRestartThread] hbase.LocalHBaseCluster(379): Waiting on 10.0.0.110,51074,1434564492618 2015-06-17 11:08:26,168 INFO [M:0;10.0.0.110:51074] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2015-06-17 11:08:26,168 INFO [M:0;10.0.0.110:51074] regionserver.HeapMemoryManager(201): Stoping HeapMemoryTuner chore. 2015-06-17 11:08:26,168 INFO [SplitLogWorker-10.0.0.110:51074] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2015-06-17 11:08:26,168 INFO [SplitLogWorker-10.0.0.110:51074] regionserver.SplitLogWorker(155): SplitLogWorker 10.0.0.110,51074,1434564492618 exiting 2015-06-17 11:08:26,169 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2015-06-17 11:08:26,168 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2015-06-17 11:08:26,168 INFO [M:0;10.0.0.110:51074] procedure2.ProcedureExecutor(490): Stopping the procedure executor 2015-06-17 11:08:26,170 INFO [M:0;10.0.0.110:51074] wal.WALProcedureStore(159): Stopping the WAL Procedure Store 2015-06-17 11:08:26,175 INFO [IPC Server handler 0 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741877_1053{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 0 2015-06-17 11:08:26,176 INFO [M:0;10.0.0.110:51074] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2015-06-17 11:08:26,176 INFO [M:0;10.0.0.110:51074] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2015-06-17 11:08:26,176 INFO [M:0;10.0.0.110:51074] regionserver.HRegionServer(1021): stopping server 10.0.0.110,51074,1434564492618 2015-06-17 11:08:26,176 DEBUG [M:0;10.0.0.110:51074] zookeeper.MetaTableLocator(617): Stopping MetaTableLocator 2015-06-17 11:08:26,176 DEBUG [RS_CLOSE_REGION-10.0.0.110:51074-0] handler.CloseRegionHandler(90): Processing close of hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:08:26,176 DEBUG [RS_CLOSE_REGION-10.0.0.110:51074-0] regionserver.HRegion(1365): Closing hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.: disabling compactions & flushes 2015-06-17 11:08:26,176 DEBUG [RS_CLOSE_REGION-10.0.0.110:51074-0] regionserver.HRegion(1392): Updates disabled for region hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:08:26,177 INFO [M:0;10.0.0.110:51074] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320023 2015-06-17 11:08:26,177 INFO [StoreCloserThread-hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.-1] regionserver.HStore(870): Closed info 2015-06-17 11:08:26,177 DEBUG [M:0;10.0.0.110:51074] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:08:26,178 INFO [M:0;10.0.0.110:51074] regionserver.CompactSplitThread(394): Waiting for Split Thread to finish... 2015-06-17 11:08:26,178 INFO [M:0;10.0.0.110:51074] regionserver.CompactSplitThread(394): Waiting for Merge Thread to finish... 2015-06-17 11:08:26,178 DEBUG [AsyncRpcChannel-pool2-t8] ipc.AsyncRpcChannel$8(574): IPC Client (-2130812629) connection to /10.0.0.110:51074 from ndimiduk.hfs.6: closed 2015-06-17 11:08:26,178 DEBUG [AsyncRpcChannel-pool2-t9] ipc.AsyncRpcChannel$8(574): IPC Client (-515746670) connection to /10.0.0.110:51015 from ndimiduk.hfs.6: closed 2015-06-17 11:08:26,178 INFO [M:0;10.0.0.110:51074] regionserver.CompactSplitThread(394): Waiting for Large Compaction Thread to finish... 2015-06-17 11:08:26,178 DEBUG [RpcServer.reader=2,bindAddress=10.0.0.110,port=51074] ipc.RpcServer$Listener(872): RpcServer.listener,port=51074: DISCONNECTING client 10.0.0.110:51094 because read count=-1. Number of active connections: 3 2015-06-17 11:08:26,178 INFO [M:0;10.0.0.110:51074] regionserver.CompactSplitThread(394): Waiting for Small Compaction Thread to finish... 2015-06-17 11:08:26,178 INFO [M:0;10.0.0.110:51074] regionserver.HRegionServer(1243): Waiting on 2 regions to close 2015-06-17 11:08:26,178 DEBUG [M:0;10.0.0.110:51074] regionserver.HRegionServer(1247): {9ccc7b1e8942a5472c9cf2d844c2fff0=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., 1588230740=hbase:meta,,1.1588230740} 2015-06-17 11:08:26,178 DEBUG [RS_CLOSE_META-10.0.0.110:51074-0] handler.CloseRegionHandler(90): Processing close of hbase:meta,,1.1588230740 2015-06-17 11:08:26,179 DEBUG [RS_CLOSE_META-10.0.0.110:51074-0] regionserver.HRegion(1365): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2015-06-17 11:08:26,179 DEBUG [RS_CLOSE_META-10.0.0.110:51074-0] regionserver.HRegion(1392): Updates disabled for region hbase:meta,,1.1588230740 2015-06-17 11:08:26,179 INFO [RS_CLOSE_META-10.0.0.110:51074-0] regionserver.HRegion(2111): Flushing 2/2 column families, memstore=1.75 KB 2015-06-17 11:08:26,182 DEBUG [RS_CLOSE_REGION-10.0.0.110:51074-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/recovered.edits/20.seqid to file, newSeqId=20, maxSeqId=18 2015-06-17 11:08:26,183 INFO [RS_CLOSE_REGION-10.0.0.110:51074-0] regionserver.HRegion(1486): Closed hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:08:26,183 DEBUG [RS_CLOSE_REGION-10.0.0.110:51074-0] handler.CloseRegionHandler(122): Closed hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:08:26,186 INFO [IPC Server handler 7 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741878_1054{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 0 2015-06-17 11:08:26,186 INFO [RS_CLOSE_META-10.0.0.110:51074-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=120000147, memsize=1.4 K, hasBloomFilter=false, into tmp file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/0cdbef2057bf48ffab39a58d41d9feda 2015-06-17 11:08:26,198 INFO [IPC Server handler 3 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741879_1055{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 0 2015-06-17 11:08:26,199 INFO [RS_CLOSE_META-10.0.0.110:51074-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=120000147, memsize=352, hasBloomFilter=false, into tmp file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/9810d8ec635a40f199322a209fb0a8f6 2015-06-17 11:08:26,205 DEBUG [RS_CLOSE_META-10.0.0.110:51074-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/0cdbef2057bf48ffab39a58d41d9feda as hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/0cdbef2057bf48ffab39a58d41d9feda 2015-06-17 11:08:26,211 INFO [RS_CLOSE_META-10.0.0.110:51074-0] regionserver.HStore(963): Added hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/0cdbef2057bf48ffab39a58d41d9feda, entries=6, sequenceid=120000147, filesize=5.3 K 2015-06-17 11:08:26,212 DEBUG [RS_CLOSE_META-10.0.0.110:51074-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/9810d8ec635a40f199322a209fb0a8f6 as hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/9810d8ec635a40f199322a209fb0a8f6 2015-06-17 11:08:26,218 INFO [RS_CLOSE_META-10.0.0.110:51074-0] regionserver.HStore(963): Added hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/9810d8ec635a40f199322a209fb0a8f6, entries=2, sequenceid=120000147, filesize=4.6 K 2015-06-17 11:08:26,219 INFO [RS_CLOSE_META-10.0.0.110:51074-0] regionserver.HRegion(2400): Finished memstore flush of ~1.75 KB/1792, currentsize=0 B/0 for region hbase:meta,,1.1588230740 in 40ms, sequenceid=120000147, compaction requested=false 2015-06-17 11:08:26,220 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(870): Closed info 2015-06-17 11:08:26,221 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(870): Closed table 2015-06-17 11:08:26,225 DEBUG [RS_CLOSE_META-10.0.0.110:51074-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/recovered.edits/120000150.seqid to file, newSeqId=120000150, maxSeqId=120000134 2015-06-17 11:08:26,226 DEBUG [RS_CLOSE_META-10.0.0.110:51074-0] coprocessor.CoprocessorHost(269): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2015-06-17 11:08:26,226 INFO [RS_CLOSE_META-10.0.0.110:51074-0] regionserver.HRegion(1486): Closed hbase:meta,,1.1588230740 2015-06-17 11:08:26,226 DEBUG [RS_CLOSE_META-10.0.0.110:51074-0] handler.CloseRegionHandler(122): Closed hbase:meta,,1.1588230740 2015-06-17 11:08:26,271 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=148, retries=350, started=42703 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:26,327 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=308, retries=350, started=75879 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:26,333 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:26,334 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51074: callId: 1365 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:26,334 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2323 ms ago, cancelled=false, msg= 2015-06-17 11:08:26,379 INFO [M:0;10.0.0.110:51074] regionserver.HRegionServer(1049): stopping server 10.0.0.110,51074,1434564492618; all regions closed. 2015-06-17 11:08:26,379 DEBUG [M:0;10.0.0.110:51074] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51074,1434564492618 2015-06-17 11:08:26,383 INFO [IPC Server handler 3 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741874_1050{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 83 2015-06-17 11:08:26,405 DEBUG [M:0;10.0.0.110:51074] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:08:26,405 INFO [M:0;10.0.0.110:51074] wal.FSHLog(977): Closed WAL: FSHLog 10.0.0.110%2C51074%2C1434564492618..meta:.meta(num 1434564492759) 2015-06-17 11:08:26,405 DEBUG [M:0;10.0.0.110:51074] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51074,1434564492618 2015-06-17 11:08:26,406 INFO [IPC Server handler 9 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741873_1049{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 83 2015-06-17 11:08:26,409 DEBUG [M:0;10.0.0.110:51074] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:08:26,409 INFO [M:0;10.0.0.110:51074] wal.FSHLog(977): Closed WAL: FSHLog 10.0.0.110%2C51074%2C1434564492618.default:(num 1434564492702) 2015-06-17 11:08:26,409 DEBUG [M:0;10.0.0.110:51074] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:08:26,409 INFO [M:0;10.0.0.110:51074] regionserver.Leases(146): M:0;10.0.0.110:51074 closing leases 2015-06-17 11:08:26,409 INFO [M:0;10.0.0.110:51074] regionserver.Leases(149): M:0;10.0.0.110:51074 closed leases 2015-06-17 11:08:26,409 INFO [M:0;10.0.0.110:51074] hbase.ChoreService(302): Chore service for: 10.0.0.110,51074,1434564492618 had [[ScheduledChore: Name: 10.0.0.110,51074,1434564492618-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.0.0.110,51074,1434564492618-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-10.0.0.110:51074 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region 10.0.0.110,51074,1434564492618 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.0.0.110,51074,1434564492618-MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS]] on shutdown 2015-06-17 11:08:26,472 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=149, retries=350, started=42904 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:26,529 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=309, retries=350, started=76081 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:26,537 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:26,538 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51074: callId: 1367 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:26,538 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2527 ms ago, cancelled=false, msg= 2015-06-17 11:08:26,651 INFO [10.0.0.110,51074,1434564492618_splitLogManager__ChoreService_1] hbase.ScheduledChore(179): Chore: SplitLogManager Timeout Monitor was stopped 2015-06-17 11:08:26,673 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=150, retries=350, started=43105 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:26,717 INFO [master//10.0.0.110:0.logRoller] regionserver.LogRoller(159): LogRoller exiting. 2015-06-17 11:08:26,719 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(146): master//10.0.0.110:0.leaseChecker closing leases 2015-06-17 11:08:26,719 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(149): master//10.0.0.110:0.leaseChecker closed leases 2015-06-17 11:08:26,731 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=310, retries=350, started=76283 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:26,742 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:26,742 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51074] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51074: callId: 1369 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51083 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:26,743 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2732 ms ago, cancelled=false, msg= 2015-06-17 11:08:26,762 INFO [RS_OPEN_META-10.0.0.110:51074-0-MetaLogRoller] regionserver.LogRoller(159): LogRoller exiting. 2015-06-17 11:08:26,765 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/10.0.0.110,51074,1434564492618 2015-06-17 11:08:26,765 INFO [M:0;10.0.0.110:51074] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320026 2015-06-17 11:08:26,766 DEBUG [M:0;10.0.0.110:51074] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:08:26,767 DEBUG [M:0;10.0.0.110:51074] master.HMaster(1074): Stopping service threads 2015-06-17 11:08:26,768 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:08:26,768 INFO [M:0;10.0.0.110:51074] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320024 2015-06-17 11:08:26,768 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:08:26,768 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(488): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:08:26,769 DEBUG [M:0;10.0.0.110:51074] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:08:26,769 DEBUG [main-EventThread] zookeeper.ZKUtil(488): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:08:26,769 INFO [M:0;10.0.0.110:51074] hbase.ChoreService(302): Chore service for: 10.0.0.110,51074,1434564492618_splitLogManager_ had [] on shutdown 2015-06-17 11:08:26,769 INFO [M:0;10.0.0.110:51074] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2015-06-17 11:08:26,769 INFO [M:0;10.0.0.110:51074] ipc.RpcServer(2219): Stopping server on 51074 2015-06-17 11:08:26,769 INFO [RpcServer.listener,port=51074] ipc.RpcServer$Listener(776): RpcServer.listener,port=51074: stopping 2015-06-17 11:08:26,770 INFO [RpcServer.responder] ipc.RpcServer$Responder(1018): RpcServer.responder: stopped 2015-06-17 11:08:26,770 INFO [RpcServer.responder] ipc.RpcServer$Responder(921): RpcServer.responder: stopping 2015-06-17 11:08:26,771 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,51074,1434564492618 2015-06-17 11:08:26,771 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,51074,1434564492618 2015-06-17 11:08:26,771 INFO [MasterRestartThread-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [10.0.0.110,51074,1434564492618] 2015-06-17 11:08:26,771 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51074-0x14e02b3a2320022, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:08:26,772 INFO [M:0;10.0.0.110:51074] regionserver.HRegionServer(1093): stopping server 10.0.0.110,51074,1434564492618; zookeeper connection closed. 2015-06-17 11:08:26,772 INFO [M:0;10.0.0.110:51074] regionserver.HRegionServer(1096): M:0;10.0.0.110:51074 exiting 2015-06-17 11:08:26,772 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(200): executing master restart 2015-06-17 11:08:26,772 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:26,772 INFO [main-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(121): /hbase/rs/10.0.0.110,51074,1434564492618 znode expired, triggering replicatorRemoved event 2015-06-17 11:08:26,772 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:08:26,773 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:26,785 INFO [MasterRestartThread] client.ConnectionUtils(105): master//10.0.0.110:0 server-side HConnection retries=350 2015-06-17 11:08:26,786 INFO [MasterRestartThread] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2015-06-17 11:08:26,788 INFO [MasterRestartThread] ipc.RpcServer$Listener(598): master//10.0.0.110:0: started 10 reader(s). 2015-06-17 11:08:26,789 INFO [MasterRestartThread] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:08:26,790 INFO [MasterRestartThread] zookeeper.RecoverableZooKeeper(121): Process identifier=master:51117 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:08:26,792 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:511170x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:08:26,793 DEBUG [MasterRestartThread] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-17 11:08:26,794 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(578): master:51117-0x14e02b3a2320027 connected 2015-06-17 11:08:26,795 DEBUG [MasterRestartThread] zookeeper.ZKUtil(488): master:51117-0x14e02b3a2320027, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:08:26,796 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:51117-0x14e02b3a2320027, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2015-06-17 11:08:26,796 INFO [RpcServer.responder] ipc.RpcServer$Responder(917): RpcServer.responder: starting 2015-06-17 11:08:26,796 INFO [RpcServer.listener,port=51117] ipc.RpcServer$Listener(729): RpcServer.listener,port=51117: starting 2015-06-17 11:08:26,796 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=0 queue=0 2015-06-17 11:08:26,796 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=1 queue=0 2015-06-17 11:08:26,796 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=2 queue=0 2015-06-17 11:08:26,797 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=3 queue=0 2015-06-17 11:08:26,797 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=4 queue=0 2015-06-17 11:08:26,797 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=0 queue=0 2015-06-17 11:08:26,797 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=1 queue=1 2015-06-17 11:08:26,797 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=2 queue=0 2015-06-17 11:08:26,797 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=3 queue=1 2015-06-17 11:08:26,797 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=4 queue=0 2015-06-17 11:08:26,797 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=5 queue=1 2015-06-17 11:08:26,798 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=6 queue=0 2015-06-17 11:08:26,798 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=7 queue=1 2015-06-17 11:08:26,798 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=8 queue=0 2015-06-17 11:08:26,798 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=9 queue=1 2015-06-17 11:08:26,798 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=10 queue=0 2015-06-17 11:08:26,798 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=11 queue=1 2015-06-17 11:08:26,798 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=12 queue=0 2015-06-17 11:08:26,798 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=13 queue=1 2015-06-17 11:08:26,798 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=14 queue=0 2015-06-17 11:08:26,798 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=15 queue=1 2015-06-17 11:08:26,799 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=16 queue=0 2015-06-17 11:08:26,799 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=17 queue=1 2015-06-17 11:08:26,799 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=18 queue=0 2015-06-17 11:08:26,799 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=19 queue=1 2015-06-17 11:08:26,799 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=0 queue=0 2015-06-17 11:08:26,799 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=1 queue=0 2015-06-17 11:08:26,799 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=2 queue=0 2015-06-17 11:08:26,800 INFO [MasterRestartThread] master.HMaster(349): hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034, hbase.cluster.distributed=false 2015-06-17 11:08:26,800 INFO [MasterRestartThread] master.HMaster(1512): Adding backup master ZNode /hbase/backup-masters/10.0.0.110,51117,1434564506788 2015-06-17 11:08:26,801 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:51117-0x14e02b3a2320027, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/10.0.0.110,51117,1434564506788 2015-06-17 11:08:26,801 INFO [M:0;10.0.0.110:51117] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x703ba76e connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:08:26,802 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51117-0x14e02b3a2320027, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:08:26,802 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:08:26,802 DEBUG [10.0.0.110:51117.activeMasterManager] zookeeper.ZKUtil(486): master:51117-0x14e02b3a2320027, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:08:26,803 INFO [10.0.0.110:51117.activeMasterManager] master.ActiveMasterManager(170): Deleting ZNode for /hbase/backup-masters/10.0.0.110,51117,1434564506788 from backup master directory 2015-06-17 11:08:26,803 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(486): master:51117-0x14e02b3a2320027, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:08:26,803 DEBUG [MasterRestartThread-EventThread] master.ActiveMasterManager(126): A master is now available 2015-06-17 11:08:26,803 DEBUG [M:0;10.0.0.110:51117-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x703ba76e0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:08:26,804 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51117-0x14e02b3a2320027, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/10.0.0.110,51117,1434564506788 2015-06-17 11:08:26,804 WARN [10.0.0.110:51117.activeMasterManager] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:08:26,804 INFO [10.0.0.110:51117.activeMasterManager] master.ActiveMasterManager(179): Registered Active Master=10.0.0.110,51117,1434564506788 2015-06-17 11:08:26,804 DEBUG [M:0;10.0.0.110:51117] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@17bf3476, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:08:26,804 DEBUG [M:0;10.0.0.110:51117] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:08:26,804 DEBUG [M:0;10.0.0.110:51117] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:08:26,805 DEBUG [M:0;10.0.0.110:51117-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x703ba76e-0x14e02b3a2320028 connected 2015-06-17 11:08:26,805 INFO [M:0;10.0.0.110:51117] regionserver.HRegionServer(774): ClusterId : b7c4b21b-aab2-44b4-8562-2b61af22ad6c 2015-06-17 11:08:26,806 DEBUG [M:0;10.0.0.110:51117] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2015-06-17 11:08:26,806 DEBUG [M:0;10.0.0.110:51117] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:08:26,807 DEBUG [M:0;10.0.0.110:51117] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2015-06-17 11:08:26,807 DEBUG [M:0;10.0.0.110:51117] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2015-06-17 11:08:26,808 DEBUG [M:0;10.0.0.110:51117] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:08:26,808 DEBUG [M:0;10.0.0.110:51117] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2015-06-17 11:08:26,809 INFO [M:0;10.0.0.110:51117] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2015-06-17 11:08:26,809 INFO [M:0;10.0.0.110:51117] compactions.PressureAwareCompactionThroughputController(175): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2015-06-17 11:08:26,809 INFO [M:0;10.0.0.110:51117] regionserver.HRegionServer$CompactionChecker(1504): CompactionChecker runs every 1sec 2015-06-17 11:08:26,809 DEBUG [M:0;10.0.0.110:51117] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@33f387c8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=10.0.0.110/10.0.0.110:0 2015-06-17 11:08:26,810 DEBUG [M:0;10.0.0.110:51117] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:08:26,810 DEBUG [M:0;10.0.0.110:51117] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:08:26,810 DEBUG [M:0;10.0.0.110:51117] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:M:0;10.0.0.110:51117 2015-06-17 11:08:26,811 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:08:26,811 DEBUG [M:0;10.0.0.110:51117] zookeeper.ZKUtil(486): master:51117-0x14e02b3a2320027, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51117,1434564506788 2015-06-17 11:08:26,811 INFO [M:0;10.0.0.110:51117] regionserver.RegionServerCoprocessorHost(65): System coprocessor loading is enabled 2015-06-17 11:08:26,811 INFO [M:0;10.0.0.110:51117] regionserver.RegionServerCoprocessorHost(66): Table coprocessor loading is enabled 2015-06-17 11:08:26,811 INFO [M:0;10.0.0.110:51117] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,51117,1434564506788 with port=51117, startcode=1434564506788 2015-06-17 11:08:26,811 DEBUG [M:0;10.0.0.110:51117] regionserver.HRegionServer(2295): Master is not running yet 2015-06-17 11:08:26,811 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:26,811 WARN [M:0;10.0.0.110:51117] regionserver.HRegionServer(900): reportForDuty failed; sleeping and then retrying. 2015-06-17 11:08:26,812 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51117,1434564506788 2015-06-17 11:08:26,815 DEBUG [10.0.0.110:51117.activeMasterManager] util.FSTableDescriptors(807): Current tableInfoPath = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2015-06-17 11:08:26,816 DEBUG [10.0.0.110:51117.activeMasterManager] util.FSTableDescriptors(811): TableInfo already exists.. Skipping creation 2015-06-17 11:08:26,820 INFO [10.0.0.110:51117.activeMasterManager] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:08:26,820 DEBUG [10.0.0.110:51117.activeMasterManager] coordination.ZKSplitLogManagerCoordination(885): Distributed log replay=true 2015-06-17 11:08:26,821 INFO [10.0.0.110:51117.activeMasterManager] coordination.ZKSplitLogManagerCoordination(599): Found 0 orphan tasks and 0 rescan nodes 2015-06-17 11:08:26,821 DEBUG [10.0.0.110:51117.activeMasterManager] util.FSTableDescriptors(222): Fetching table descriptors from the filesystem. 2015-06-17 11:08:26,829 INFO [10.0.0.110:51117.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x6884cd7e connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:08:26,831 DEBUG [10.0.0.110:51117.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x6884cd7e0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:08:26,832 DEBUG [10.0.0.110:51117.activeMasterManager] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b912b9c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:08:26,832 DEBUG [10.0.0.110:51117.activeMasterManager] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:08:26,832 DEBUG [10.0.0.110:51117.activeMasterManager] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:08:26,833 DEBUG [10.0.0.110:51117.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x6884cd7e-0x14e02b3a2320029 connected 2015-06-17 11:08:26,834 INFO [10.0.0.110:51117.activeMasterManager] balancer.StochasticLoadBalancer(135): loading config 2015-06-17 11:08:26,834 DEBUG [10.0.0.110:51117.activeMasterManager] zookeeper.ZKUtil(488): master:51117-0x14e02b3a2320027, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2015-06-17 11:08:26,834 DEBUG [10.0.0.110:51117.activeMasterManager] zookeeper.ZKUtil(486): master:51117-0x14e02b3a2320027, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:26,835 DEBUG [10.0.0.110:51117.activeMasterManager] zookeeper.ZKUtil(486): master:51117-0x14e02b3a2320027, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51117,1434564506788 2015-06-17 11:08:26,835 DEBUG [10.0.0.110:51117.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:26,835 DEBUG [10.0.0.110:51117.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,51117,1434564506788 2015-06-17 11:08:26,836 INFO [10.0.0.110:51117.activeMasterManager] master.HMaster(568): Server active/primary master=10.0.0.110,51117,1434564506788, sessionid=0x14e02b3a2320027, setting cluster-up flag (Was=true) 2015-06-17 11:08:26,837 DEBUG [10.0.0.110:51117.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:08:26,837 INFO [10.0.0.110:51117.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2015-06-17 11:08:26,838 DEBUG [10.0.0.110:51117.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,51117,1434564506788 2015-06-17 11:08:26,839 DEBUG [10.0.0.110:51117.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:08:26,839 INFO [10.0.0.110:51117.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2015-06-17 11:08:26,840 DEBUG [10.0.0.110:51117.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,51117,1434564506788 2015-06-17 11:08:26,840 INFO [10.0.0.110:51117.activeMasterManager] master.MasterCoprocessorHost(86): System coprocessor loading is enabled 2015-06-17 11:08:26,840 DEBUG [10.0.0.110:51117.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_OPEN_REGION-10.0.0.110:51117, corePoolSize=5, maxPoolSize=5 2015-06-17 11:08:26,840 DEBUG [10.0.0.110:51117.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_CLOSE_REGION-10.0.0.110:51117, corePoolSize=5, maxPoolSize=5 2015-06-17 11:08:26,840 DEBUG [10.0.0.110:51117.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_SERVER_OPERATIONS-10.0.0.110:51117, corePoolSize=5, maxPoolSize=5 2015-06-17 11:08:26,840 DEBUG [10.0.0.110:51117.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_META_SERVER_OPERATIONS-10.0.0.110:51117, corePoolSize=5, maxPoolSize=5 2015-06-17 11:08:26,840 DEBUG [10.0.0.110:51117.activeMasterManager] executor.ExecutorService(91): Starting executor service name=M_LOG_REPLAY_OPS-10.0.0.110:51117, corePoolSize=10, maxPoolSize=10 2015-06-17 11:08:26,841 DEBUG [10.0.0.110:51117.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_TABLE_OPERATIONS-10.0.0.110:51117, corePoolSize=1, maxPoolSize=1 2015-06-17 11:08:26,841 INFO [10.0.0.110:51117.activeMasterManager] procedure2.ProcedureExecutor(446): Starting procedure executor threads=9 2015-06-17 11:08:26,841 INFO [10.0.0.110:51117.activeMasterManager] wal.WALProcedureStore(214): Starting WAL Procedure Store lease recovery 2015-06-17 11:08:26,842 INFO [10.0.0.110:51117.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log 2015-06-17 11:08:26,842 INFO [10.0.0.110:51117.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log after 0ms 2015-06-17 11:08:26,842 DEBUG [10.0.0.110:51117.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log; isDirectory=false; length=14915; replication=1; blocksize=134217728; modification_time=1434564413352; access_time=1434564405122; owner=ndimiduk; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:08:26,844 INFO [10.0.0.110:51117.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000007.log 2015-06-17 11:08:26,844 INFO [10.0.0.110:51117.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000007.log after 0ms 2015-06-17 11:08:26,845 DEBUG [10.0.0.110:51117.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000007.log; isDirectory=false; length=60; replication=1; blocksize=134217728; modification_time=1434564506175; access_time=1434564492682; owner=ndimiduk.hfs.6; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:08:26,848 DEBUG [10.0.0.110:51117.activeMasterManager] wal.WALProcedureStore(610): Roll new state log: 8 2015-06-17 11:08:26,850 INFO [10.0.0.110:51117.activeMasterManager] wal.WALProcedureStore(237): Lease acquired for flushLogId: 8 2015-06-17 11:08:26,850 INFO [10.0.0.110:51117.activeMasterManager] wal.ProcedureWALFormatReader(150): No active entry found in state log hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000007.log. removing it 2015-06-17 11:08:26,852 DEBUG [10.0.0.110:51117.activeMasterManager] procedure2.ProcedureExecutor$1(278): load procedures maxProcId=7 2015-06-17 11:08:26,853 DEBUG [10.0.0.110:51117.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=RUNNABLE isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=7 state=RUNNABLE 2015-06-17 11:08:26,853 DEBUG [10.0.0.110:51117.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DeleteTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=6 state=FINISHED 2015-06-17 11:08:26,853 DEBUG [10.0.0.110:51117.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:26,853 DEBUG [10.0.0.110:51117.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=5 state=FINISHED 2015-06-17 11:08:26,853 DEBUG [10.0.0.110:51117.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:26,853 DEBUG [10.0.0.110:51117.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: EnableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=4 state=FINISHED 2015-06-17 11:08:26,853 DEBUG [10.0.0.110:51117.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:26,853 DEBUG [10.0.0.110:51117.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=3 state=FINISHED 2015-06-17 11:08:26,853 DEBUG [10.0.0.110:51117.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:26,853 DEBUG [10.0.0.110:51117.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=2 state=FINISHED 2015-06-17 11:08:26,854 DEBUG [10.0.0.110:51117.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:26,854 DEBUG [10.0.0.110:51117.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=hbase:namespace) user=ndimiduk (auth:SIMPLE) id=1 state=FINISHED 2015-06-17 11:08:26,854 DEBUG [10.0.0.110:51117.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:26,854 DEBUG [10.0.0.110:51117.activeMasterManager] wal.WALProcedureStore(652): Remove log: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000007.log 2015-06-17 11:08:26,854 INFO [IPC Server handler 2 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741877_1053 127.0.0.1:50896 2015-06-17 11:08:26,856 DEBUG [10.0.0.110:51117.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2015-06-17 11:08:26,856 INFO [10.0.0.110:51117.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:08:26,858 DEBUG [10.0.0.110:51117.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:08:26,858 DEBUG [10.0.0.110:51117.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/replication/rs already exists 2015-06-17 11:08:26,858 DEBUG [10.0.0.110:51117.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2015-06-17 11:08:26,859 DEBUG [10.0.0.110:51117.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner 2015-06-17 11:08:26,859 DEBUG [10.0.0.110:51117.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): replicationLogCleaner-0x14e02b3a232002a connected 2015-06-17 11:08:26,859 DEBUG [10.0.0.110:51117.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2015-06-17 11:08:26,860 DEBUG [10.0.0.110:51117.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2015-06-17 11:08:26,860 DEBUG [10.0.0.110:51117.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2015-06-17 11:08:26,860 INFO [10.0.0.110:51117.activeMasterManager] master.ServerManager(1011): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 1, timeout of 4500 ms, interval of 1500 ms. 2015-06-17 11:08:26,860 INFO [M:0;10.0.0.110:51117] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,51117,1434564506788 with port=51117, startcode=1434564506788 2015-06-17 11:08:26,860 INFO [M:0;10.0.0.110:51117] master.ServerManager(459): Registering server=10.0.0.110,51117,1434564506788 2015-06-17 11:08:26,860 INFO [M:0;10.0.0.110:51117] regionserver.HRegionServer(1341): Config from master: hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034 2015-06-17 11:08:26,860 INFO [M:0;10.0.0.110:51117] regionserver.HRegionServer(1341): Config from master: fs.defaultFS=hdfs://localhost:50895 2015-06-17 11:08:26,860 INFO [M:0;10.0.0.110:51117] regionserver.HRegionServer(1341): Config from master: hbase.master.info.port=-1 2015-06-17 11:08:26,860 WARN [M:0;10.0.0.110:51117] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:08:26,860 INFO [M:0;10.0.0.110:51117] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:08:26,861 DEBUG [M:0;10.0.0.110:51117] regionserver.HRegionServer(1603): logdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51117,1434564506788 2015-06-17 11:08:26,862 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=350, started=13847 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:26,863 DEBUG [M:0;10.0.0.110:51117] regionserver.Replication(142): ReplicationStatisticsThread 300 2015-06-17 11:08:26,863 INFO [M:0;10.0.0.110:51117] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:08:26,865 INFO [M:0;10.0.0.110:51117] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.0.0.110%2C51117%2C1434564506788.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51117,1434564506788, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:08:26,871 INFO [M:0;10.0.0.110:51117] wal.FSHLog(1365): Slow sync cost: 4 ms, current pipeline: [] 2015-06-17 11:08:26,872 INFO [M:0;10.0.0.110:51117] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51117,1434564506788/10.0.0.110%2C51117%2C1434564506788.default.1434564506865 2015-06-17 11:08:26,872 INFO [M:0;10.0.0.110:51117] regionserver.MetricsRegionServerWrapperImpl(113): Computing regionserver metrics every 5000 milliseconds 2015-06-17 11:08:26,873 DEBUG [M:0;10.0.0.110:51117] executor.ExecutorService(91): Starting executor service name=RS_OPEN_REGION-10.0.0.110:51117, corePoolSize=3, maxPoolSize=3 2015-06-17 11:08:26,873 DEBUG [M:0;10.0.0.110:51117] executor.ExecutorService(91): Starting executor service name=RS_OPEN_META-10.0.0.110:51117, corePoolSize=1, maxPoolSize=1 2015-06-17 11:08:26,873 DEBUG [M:0;10.0.0.110:51117] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_REGION-10.0.0.110:51117, corePoolSize=3, maxPoolSize=3 2015-06-17 11:08:26,873 DEBUG [M:0;10.0.0.110:51117] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_META-10.0.0.110:51117, corePoolSize=1, maxPoolSize=1 2015-06-17 11:08:26,873 DEBUG [M:0;10.0.0.110:51117] executor.ExecutorService(91): Starting executor service name=RS_LOG_REPLAY_OPS-10.0.0.110:51117, corePoolSize=2, maxPoolSize=2 2015-06-17 11:08:26,873 DEBUG [M:0;10.0.0.110:51117] executor.ExecutorService(91): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-10.0.0.110:51117, corePoolSize=3, maxPoolSize=3 2015-06-17 11:08:26,875 DEBUG [M:0;10.0.0.110:51117] zookeeper.ZKUtil(486): master:51117-0x14e02b3a2320027, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:26,875 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=151, retries=350, started=43307 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:26,875 DEBUG [M:0;10.0.0.110:51117] zookeeper.ZKUtil(486): master:51117-0x14e02b3a2320027, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51117,1434564506788 2015-06-17 11:08:26,875 INFO [M:0;10.0.0.110:51117] regionserver.ReplicationSourceManager(229): Current list of replicators: [10.0.0.110,50905,1434564404764, 10.0.0.110,51117,1434564506788] other RSs: [10.0.0.110,50905,1434564404764, 10.0.0.110,51117,1434564506788] 2015-06-17 11:08:26,888 INFO [M:0;10.0.0.110:51117] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x4c946540 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:08:26,890 DEBUG [M:0;10.0.0.110:51117-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x4c9465400x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:08:26,891 DEBUG [M:0;10.0.0.110:51117] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodecWithTags@67c7764f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:08:26,891 DEBUG [M:0;10.0.0.110:51117] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:08:26,891 DEBUG [M:0;10.0.0.110:51117] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:08:26,891 DEBUG [M:0;10.0.0.110:51117-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x4c946540-0x14e02b3a232002b connected 2015-06-17 11:08:26,904 INFO [M:0;10.0.0.110:51117] regionserver.HeapMemoryManager(190): Starting HeapMemoryTuner chore. 2015-06-17 11:08:26,904 INFO [SplitLogWorker-10.0.0.110:51117] regionserver.SplitLogWorker(134): SplitLogWorker 10.0.0.110,51117,1434564506788 starting 2015-06-17 11:08:26,904 INFO [M:0;10.0.0.110:51117] regionserver.HRegionServer(1363): Serving as 10.0.0.110,51117,1434564506788, RpcServer on 10.0.0.110/10.0.0.110:51117, sessionid=0x14e02b3a2320027 2015-06-17 11:08:26,904 DEBUG [M:0;10.0.0.110:51117] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2015-06-17 11:08:26,904 DEBUG [M:0;10.0.0.110:51117] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager 10.0.0.110,51117,1434564506788 2015-06-17 11:08:26,904 DEBUG [M:0;10.0.0.110:51117] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,51117,1434564506788' 2015-06-17 11:08:26,905 DEBUG [M:0;10.0.0.110:51117] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2015-06-17 11:08:26,905 DEBUG [M:0;10.0.0.110:51117] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2015-06-17 11:08:26,905 DEBUG [M:0;10.0.0.110:51117] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2015-06-17 11:08:26,905 DEBUG [M:0;10.0.0.110:51117] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2015-06-17 11:08:26,905 DEBUG [M:0;10.0.0.110:51117] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager 10.0.0.110,51117,1434564506788 2015-06-17 11:08:26,905 DEBUG [M:0;10.0.0.110:51117] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,51117,1434564506788' 2015-06-17 11:08:26,905 DEBUG [M:0;10.0.0.110:51117] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2015-06-17 11:08:26,906 DEBUG [M:0;10.0.0.110:51117] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2015-06-17 11:08:26,906 DEBUG [M:0;10.0.0.110:51117] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2015-06-17 11:08:26,906 INFO [M:0;10.0.0.110:51117] quotas.RegionServerQuotaManager(62): Quota support disabled 2015-06-17 11:08:26,911 INFO [10.0.0.110:51117.activeMasterManager] master.ServerManager(1028): Finished waiting for region servers count to settle; checked in 1, slept for 51 ms, expecting minimum of 1, maximum of 1, master is running 2015-06-17 11:08:26,911 INFO [10.0.0.110:51117.activeMasterManager] master.ServerManager(459): Registering server=10.0.0.110,50905,1434564404764 2015-06-17 11:08:26,912 INFO [10.0.0.110:51117.activeMasterManager] master.HMaster(665): Registered server found up in zk but who has not yet reported in: 10.0.0.110,50905,1434564404764 2015-06-17 11:08:26,914 INFO [10.0.0.110:51117.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50905,1434564404764 belongs to an existing region server 2015-06-17 11:08:26,915 INFO [10.0.0.110:51117.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51117,1434564506788 belongs to an existing region server 2015-06-17 11:08:26,917 WARN [10.0.0.110:51117.activeMasterManager] master.RegionStates(339): Open region's last host null should be the same as the current one 10.0.0.110,51074,1434564492618, ignored the last and used the current one 2015-06-17 11:08:26,932 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=311, retries=350, started=76484 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:26,944 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:08:26,946 DEBUG [AsyncRpcChannel-pool2-t12] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service MasterService, sasl=false 2015-06-17 11:08:26,946 DEBUG [RpcServer.listener,port=51117] ipc.RpcServer$Listener(840): RpcServer.listener,port=51117: connection from 10.0.0.110:51126; # active connections: 1 2015-06-17 11:08:26,947 INFO [RpcServer.reader=1,bindAddress=10.0.0.110,port=51117] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51126 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:08:26,948 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:26,948 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51117: callId: 1372 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:26,948 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2937 ms ago, cancelled=false, msg= 2015-06-17 11:08:27,063 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=350, started=14048 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:27,075 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=152, retries=350, started=43507 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:27,132 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=312, retries=350, started=76684 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:27,149 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:27,150 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51117: callId: 1374 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:27,150 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3139 ms ago, cancelled=false, msg= 2015-06-17 11:08:27,266 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=350, started=14251 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:27,276 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=153, retries=350, started=43708 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:27,333 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=313, retries=350, started=76885 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:27,352 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:27,353 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51117: callId: 1376 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:27,353 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3342 ms ago, cancelled=false, msg= 2015-06-17 11:08:27,466 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=350, started=14451 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:27,478 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=154, retries=350, started=43910 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:27,535 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=314, retries=350, started=77087 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:27,554 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:27,554 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51117: callId: 1378 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:27,555 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3544 ms ago, cancelled=false, msg= 2015-06-17 11:08:27,667 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=350, started=14652 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:27,679 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=155, retries=350, started=44111 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:27,737 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=315, retries=350, started=77289 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:27,757 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:27,757 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51117: callId: 1380 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:27,760 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3749 ms ago, cancelled=false, msg= 2015-06-17 11:08:27,868 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=350, started=14853 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:27,880 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=156, retries=350, started=44312 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:27,939 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=316, retries=350, started=77491 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:27,962 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:27,962 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51117: callId: 1382 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:27,963 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3952 ms ago, cancelled=false, msg= 2015-06-17 11:08:28,068 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=350, started=15053 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:28,081 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=157, retries=350, started=44513 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:28,140 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=317, retries=350, started=77692 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:28,166 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:28,166 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51117: callId: 1384 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:28,166 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4155 ms ago, cancelled=false, msg= 2015-06-17 11:08:28,269 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=350, started=15254 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:28,282 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=158, retries=350, started=44714 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:28,341 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=318, retries=350, started=77893 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:28,368 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:28,369 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51117: callId: 1386 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:28,369 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4358 ms ago, cancelled=false, msg= 2015-06-17 11:08:28,470 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=350, started=15455 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:28,485 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=159, retries=350, started=44917 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:28,514 DEBUG [AsyncRpcChannel-pool2-t13] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2015-06-17 11:08:28,514 DEBUG [RpcServer.listener,port=51117] ipc.RpcServer$Listener(840): RpcServer.listener,port=51117: connection from 10.0.0.110:51127; # active connections: 2 2015-06-17 11:08:28,515 INFO [RpcServer.reader=2,bindAddress=10.0.0.110,port=51117] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51127 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:08:28,542 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=319, retries=350, started=78094 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:28,571 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:28,571 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51117: callId: 1388 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:28,572 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4561 ms ago, cancelled=false, msg= 2015-06-17 11:08:28,671 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=350, started=15656 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:28,686 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=160, retries=350, started=45118 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:28,743 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=320, retries=350, started=78295 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:28,774 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:28,774 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51117: callId: 1390 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:28,775 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4764 ms ago, cancelled=false, msg= 2015-06-17 11:08:28,873 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=350, started=15858 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:28,886 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=161, retries=350, started=45318 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:28,944 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=321, retries=350, started=78496 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:28,978 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:28,978 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51117: callId: 1392 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:28,978 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4967 ms ago, cancelled=false, msg= 2015-06-17 11:08:29,074 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=350, started=16059 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:29,087 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=162, retries=350, started=45519 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:29,145 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=322, retries=350, started=78697 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:29,180 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:29,180 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51117: callId: 1394 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:29,181 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5170 ms ago, cancelled=false, msg= 2015-06-17 11:08:29,276 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=350, started=16261 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:29,288 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=163, retries=350, started=45720 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:29,345 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=323, retries=350, started=78897 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:29,382 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:29,382 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51117: callId: 1396 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:29,382 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5371 ms ago, cancelled=false, msg= 2015-06-17 11:08:29,383 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564504011, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:08:29,476 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=350, started=16461 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:29,490 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=164, retries=350, started=45922 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:29,546 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=324, retries=350, started=79098 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:29,585 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:29,585 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51117: callId: 1398 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:29,588 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:29,588 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51117: callId: 1400 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:29,592 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:29,592 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51117: callId: 1402 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:29,598 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:29,598 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51117: callId: 1404 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:29,609 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:29,609 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51117: callId: 1406 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:29,630 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:29,630 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51117: callId: 1408 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:29,672 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:29,672 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51117: callId: 1410 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:29,677 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=350, started=16662 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:29,691 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=165, retries=350, started=46123 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:29,747 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=325, retries=350, started=79299 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:29,774 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:29,774 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51117: callId: 1412 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:29,875 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:29,875 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51117: callId: 1414 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:29,877 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=350, started=16862 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:29,891 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=166, retries=350, started=46323 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:29,948 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=326, retries=350, started=79500 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:29,978 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:29,978 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51117: callId: 1416 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:30,079 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=350, started=17064 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:30,079 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:30,079 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51117: callId: 1418 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:30,079 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=495 ms ago, cancelled=false, msg= 2015-06-17 11:08:30,092 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=167, retries=350, started=46524 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:30,149 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=327, retries=350, started=79701 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:30,281 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=350, started=17266 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:30,282 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:30,282 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51117: callId: 1420 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:30,282 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=698 ms ago, cancelled=false, msg= 2015-06-17 11:08:30,292 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=168, retries=350, started=46724 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:30,350 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=328, retries=350, started=79902 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:30,481 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=350, started=17466 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:30,483 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:30,483 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51117: callId: 1422 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:30,484 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=900 ms ago, cancelled=false, msg= 2015-06-17 11:08:30,493 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=169, retries=350, started=46925 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:30,552 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=329, retries=350, started=80104 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:30,683 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=350, started=17668 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:30,685 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:30,685 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51117: callId: 1424 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:30,686 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1102 ms ago, cancelled=false, msg= 2015-06-17 11:08:30,693 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=170, retries=350, started=47125 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:30,752 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=330, retries=350, started=80304 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:30,883 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=350, started=17868 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:30,887 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:30,887 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51117: callId: 1426 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:30,887 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1303 ms ago, cancelled=false, msg= 2015-06-17 11:08:30,894 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=171, retries=350, started=47326 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:30,953 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=331, retries=350, started=80505 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:31,085 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=350, started=18070 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:31,090 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:31,090 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51117: callId: 1428 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:31,090 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1506 ms ago, cancelled=false, msg= 2015-06-17 11:08:31,095 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=172, retries=350, started=47527 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:31,155 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=332, retries=350, started=80707 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:31,286 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=350, started=18271 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:31,293 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:31,293 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51117: callId: 1430 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:31,293 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1709 ms ago, cancelled=false, msg= 2015-06-17 11:08:31,296 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=173, retries=350, started=47728 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:31,356 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=333, retries=350, started=80907 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:31,487 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=350, started=18472 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:31,495 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:31,495 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51117: callId: 1432 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:31,495 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1911 ms ago, cancelled=false, msg= 2015-06-17 11:08:31,498 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=174, retries=350, started=47930 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:31,557 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=334, retries=350, started=81109 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:31,689 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=350, started=18674 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:31,696 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:31,696 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51117: callId: 1434 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:31,697 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2113 ms ago, cancelled=false, msg= 2015-06-17 11:08:31,698 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=175, retries=350, started=48130 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:31,759 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=335, retries=350, started=81311 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:31,891 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=35, retries=350, started=18876 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:31,898 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=176, retries=350, started=48330 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:31,899 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:31,899 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51117: callId: 1436 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:31,899 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2315 ms ago, cancelled=false, msg= 2015-06-17 11:08:31,960 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=336, retries=350, started=81512 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:32,093 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=36, retries=350, started=19078 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:32,099 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=177, retries=350, started=48531 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:32,101 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:32,102 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51117: callId: 1438 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:32,102 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2518 ms ago, cancelled=false, msg= 2015-06-17 11:08:32,161 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=337, retries=350, started=81712 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:32,293 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=37, retries=350, started=19278 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:32,301 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=178, retries=350, started=48733 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:32,303 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:32,303 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51117: callId: 1440 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:32,303 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2719 ms ago, cancelled=false, msg= 2015-06-17 11:08:32,362 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=338, retries=350, started=81914 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:32,496 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=38, retries=350, started=19481 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:32,502 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=179, retries=350, started=48934 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:32,505 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:32,505 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51117: callId: 1442 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:32,506 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2922 ms ago, cancelled=false, msg= 2015-06-17 11:08:32,563 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=339, retries=350, started=82115 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:32,696 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=39, retries=350, started=19681 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:32,703 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=180, retries=350, started=49135 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:32,707 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:32,708 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51117: callId: 1444 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:32,708 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3124 ms ago, cancelled=false, msg= 2015-06-17 11:08:32,763 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=340, retries=350, started=82315 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:32,898 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=40, retries=350, started=19883 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:32,905 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=181, retries=350, started=49337 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:32,910 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:32,910 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51117: callId: 1446 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:32,910 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3326 ms ago, cancelled=false, msg= 2015-06-17 11:08:32,963 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=341, retries=350, started=82515 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:33,098 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=41, retries=350, started=20083 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:33,105 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=182, retries=350, started=49537 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:33,112 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:33,112 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51117: callId: 1448 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:33,113 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3529 ms ago, cancelled=false, msg= 2015-06-17 11:08:33,165 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=342, retries=350, started=82716 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:33,298 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=42, retries=350, started=20283 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:33,307 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=183, retries=350, started=49739 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:33,316 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:33,316 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51117: callId: 1450 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:33,316 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3732 ms ago, cancelled=false, msg= 2015-06-17 11:08:33,366 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=343, retries=350, started=82918 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:33,500 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=43, retries=350, started=20485 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:33,509 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=184, retries=350, started=49941 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:33,519 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:33,519 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51117: callId: 1452 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:33,520 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3936 ms ago, cancelled=false, msg= 2015-06-17 11:08:33,567 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=344, retries=350, started=83119 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:33,702 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=44, retries=350, started=20687 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:33,709 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=185, retries=350, started=50141 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:33,723 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:33,723 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51117: callId: 1454 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:33,723 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4139 ms ago, cancelled=false, msg= 2015-06-17 11:08:33,767 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=345, retries=350, started=83319 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:33,902 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=45, retries=350, started=20887 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:33,910 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=186, retries=350, started=50342 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:33,925 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:33,925 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51117: callId: 1456 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:33,925 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4341 ms ago, cancelled=false, msg= 2015-06-17 11:08:33,968 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=346, retries=350, started=83520 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:34,103 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=46, retries=350, started=21088 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:34,111 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=187, retries=350, started=50543 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:34,128 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:34,128 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51117: callId: 1458 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:34,129 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4545 ms ago, cancelled=false, msg= 2015-06-17 11:08:34,168 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=347, retries=350, started=83720 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:34,305 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=47, retries=350, started=21290 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:34,312 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=188, retries=350, started=50744 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:34,333 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:34,333 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51117: callId: 1460 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:34,333 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4749 ms ago, cancelled=false, msg= 2015-06-17 11:08:34,369 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=348, retries=350, started=83921 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:34,507 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=48, retries=350, started=21492 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:34,512 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=189, retries=350, started=50944 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:34,534 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:34,534 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51117: callId: 1462 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:34,535 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4951 ms ago, cancelled=false, msg= 2015-06-17 11:08:34,570 INFO [10.0.0.110:50975.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=349, retries=350, started=84122 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50903,1434564404489, seqNum=2 2015-06-17 11:08:34,574 WARN [10.0.0.110:50975.activeMasterManager] master.TableNamespaceManager(292): Caught exception in initializing namespace table manager org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=350, exceptions: Wed Jun 17 11:07:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call id=33 on server /10.0.0.110:50903 aborted: connection is closing Wed Jun 17 11:07:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:721) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:687) at org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:137) at org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:268) at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:98) at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:919) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:749) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1542) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: hconnection-0x491e693a closed at org.apache.hadoop.hbase.client.ConnectionImplementation.getTableState(ConnectionImplementation.java:2235) at org.apache.hadoop.hbase.client.ConnectionImplementation.isTableDisabled(ConnectionImplementation.java:553) at org.apache.hadoop.hbase.client.ConnectionImplementation.relocateRegion(ConnectionImplementation.java:718) at org.apache.hadoop.hbase.client.ConnectionImplementation.relocateRegion(ConnectionImplementation.java:706) at org.apache.hadoop.hbase.client.ConnectionImplementation.getRegionLocation(ConnectionImplementation.java:531) at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:81) at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:79) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) ... 10 more 2015-06-17 11:08:34,575 FATAL [10.0.0.110:50975.activeMasterManager] master.HMaster$1(1546): Failed to become active master org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=350, exceptions: Wed Jun 17 11:07:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call id=33 on server /10.0.0.110:50903 aborted: connection is closing Wed Jun 17 11:07:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564430448, pause=1, retries=350}, java.io.IOException: hconnection-0x491e693a closed at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:721) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:687) at org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:137) at org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:268) at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:98) at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:919) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:749) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1542) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: hconnection-0x491e693a closed at org.apache.hadoop.hbase.client.ConnectionImplementation.getTableState(ConnectionImplementation.java:2235) at org.apache.hadoop.hbase.client.ConnectionImplementation.isTableDisabled(ConnectionImplementation.java:553) at org.apache.hadoop.hbase.client.ConnectionImplementation.relocateRegion(ConnectionImplementation.java:718) at org.apache.hadoop.hbase.client.ConnectionImplementation.relocateRegion(ConnectionImplementation.java:706) at org.apache.hadoop.hbase.client.ConnectionImplementation.getRegionLocation(ConnectionImplementation.java:531) at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:81) at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:79) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) ... 10 more 2015-06-17 11:08:34,709 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=49, retries=350, started=21694 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:34,713 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=190, retries=350, started=51145 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:34,736 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:34,736 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51117: callId: 1464 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:34,736 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5152 ms ago, cancelled=false, msg= 2015-06-17 11:08:34,909 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=50, retries=350, started=21894 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:34,914 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=191, retries=350, started=51346 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:34,937 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:34,937 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51117: callId: 1466 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:34,938 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5354 ms ago, cancelled=false, msg= 2015-06-17 11:08:34,938 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564509584, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:08:35,110 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=51, retries=350, started=22095 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:35,115 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=192, retries=350, started=51547 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:35,140 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:35,140 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51117: callId: 1468 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:35,143 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:35,143 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51117: callId: 1470 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:35,147 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:35,147 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51117: callId: 1472 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:35,154 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:35,154 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51117: callId: 1474 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:35,165 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:35,165 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51117: callId: 1476 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:35,186 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:35,186 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51117: callId: 1478 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:35,227 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:35,227 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51117: callId: 1480 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:35,310 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=52, retries=350, started=22295 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:35,315 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=193, retries=350, started=51747 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:35,329 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:35,329 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51117: callId: 1482 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:35,431 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:35,431 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51117: callId: 1484 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:35,513 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=53, retries=350, started=22498 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:35,516 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=194, retries=350, started=51948 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:35,533 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:35,533 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51117: callId: 1486 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:35,635 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:35,635 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51117: callId: 1488 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:35,635 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=496 ms ago, cancelled=false, msg= 2015-06-17 11:08:35,714 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=54, retries=350, started=22699 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:35,717 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=195, retries=350, started=52149 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:35,837 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:35,837 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51117: callId: 1490 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:35,838 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=699 ms ago, cancelled=false, msg= 2015-06-17 11:08:35,915 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=55, retries=350, started=22900 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:35,918 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=196, retries=350, started=52350 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:36,040 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:36,040 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51117: callId: 1492 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:36,040 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=901 ms ago, cancelled=false, msg= 2015-06-17 11:08:36,118 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=56, retries=350, started=23103 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:36,120 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=197, retries=350, started=52552 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:36,242 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:36,242 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51117: callId: 1494 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:36,242 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1103 ms ago, cancelled=false, msg= 2015-06-17 11:08:36,318 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=57, retries=350, started=23303 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:36,321 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=198, retries=350, started=52753 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:36,445 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:36,445 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51117: callId: 1496 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:36,446 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1307 ms ago, cancelled=false, msg= 2015-06-17 11:08:36,518 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=58, retries=350, started=23503 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:36,522 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=199, retries=350, started=52954 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:36,648 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:36,648 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51117: callId: 1498 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:36,648 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1509 ms ago, cancelled=false, msg= 2015-06-17 11:08:36,718 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=59, retries=350, started=23703 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:36,723 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=200, retries=350, started=53155 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:36,829 DEBUG [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(188): Sleeping for 3898 before executing next master restart count=7 2015-06-17 11:08:36,850 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:36,850 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51117: callId: 1500 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:36,850 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1711 ms ago, cancelled=false, msg= 2015-06-17 11:08:36,920 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=60, retries=350, started=23905 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:36,925 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=201, retries=350, started=53357 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:37,053 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:37,053 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51117: callId: 1502 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:37,054 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1915 ms ago, cancelled=false, msg= 2015-06-17 11:08:37,120 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=61, retries=350, started=24105 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:37,125 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=202, retries=350, started=53557 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:37,256 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:37,256 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51117: callId: 1504 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:37,256 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2117 ms ago, cancelled=false, msg= 2015-06-17 11:08:37,322 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=62, retries=350, started=24306 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:37,325 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=203, retries=350, started=53757 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:37,457 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:37,457 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51117: callId: 1506 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:37,458 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2319 ms ago, cancelled=false, msg= 2015-06-17 11:08:37,523 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=63, retries=350, started=24508 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:37,525 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=204, retries=350, started=53957 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:37,660 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:37,660 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51117: callId: 1508 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:37,660 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2521 ms ago, cancelled=false, msg= 2015-06-17 11:08:37,723 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=64, retries=350, started=24708 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:37,725 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=205, retries=350, started=54157 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:37,863 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:37,863 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51117: callId: 1510 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:37,864 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2725 ms ago, cancelled=false, msg= 2015-06-17 11:08:37,923 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=65, retries=350, started=24908 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:37,926 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=206, retries=350, started=54358 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:38,066 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:38,066 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51117: callId: 1512 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:38,067 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2928 ms ago, cancelled=false, msg= 2015-06-17 11:08:38,123 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=66, retries=350, started=25108 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:38,127 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=207, retries=350, started=54559 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:38,268 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:38,268 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51117: callId: 1514 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:38,268 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3129 ms ago, cancelled=false, msg= 2015-06-17 11:08:38,326 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=67, retries=350, started=25310 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:38,327 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=208, retries=350, started=54759 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:38,470 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:38,470 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51117: callId: 1516 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:38,470 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3331 ms ago, cancelled=false, msg= 2015-06-17 11:08:38,528 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=68, retries=350, started=25513 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:38,528 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=209, retries=350, started=54960 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:38,674 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:38,674 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51117: callId: 1518 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:38,674 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3535 ms ago, cancelled=false, msg= 2015-06-17 11:08:38,729 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=69, retries=350, started=25714 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:38,729 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=210, retries=350, started=55161 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:38,877 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:38,877 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51117: callId: 1520 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:38,877 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3738 ms ago, cancelled=false, msg= 2015-06-17 11:08:38,929 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=211, retries=350, started=55361 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:38,930 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=70, retries=350, started=25915 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:39,079 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:39,079 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51117: callId: 1522 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:39,079 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3940 ms ago, cancelled=false, msg= 2015-06-17 11:08:39,130 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=212, retries=350, started=55561 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:39,131 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=71, retries=350, started=26116 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:39,282 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:39,282 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51117: callId: 1524 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:39,283 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4144 ms ago, cancelled=false, msg= 2015-06-17 11:08:39,330 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=213, retries=350, started=55762 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:39,332 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=72, retries=350, started=26317 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:39,484 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:39,484 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51117: callId: 1526 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:39,484 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4345 ms ago, cancelled=false, msg= 2015-06-17 11:08:39,530 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=214, retries=350, started=55962 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:39,534 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=73, retries=350, started=26519 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:39,687 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:39,687 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51117: callId: 1528 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:39,688 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4549 ms ago, cancelled=false, msg= 2015-06-17 11:08:39,731 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=215, retries=350, started=56163 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:39,734 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=74, retries=350, started=26719 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:39,889 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:39,889 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51117: callId: 1530 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:39,889 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4750 ms ago, cancelled=false, msg= 2015-06-17 11:08:39,932 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=216, retries=350, started=56364 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:39,935 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=75, retries=350, started=26920 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:40,092 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:40,092 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51117: callId: 1532 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:40,093 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4954 ms ago, cancelled=false, msg= 2015-06-17 11:08:40,134 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=217, retries=350, started=56566 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:40,135 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=76, retries=350, started=27120 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:40,295 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:40,295 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51117: callId: 1534 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:40,295 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5156 ms ago, cancelled=false, msg= 2015-06-17 11:08:40,336 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=218, retries=350, started=56768 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:40,336 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=77, retries=350, started=27321 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:40,496 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:40,496 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51117: callId: 1536 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:40,497 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5357 ms ago, cancelled=false, msg= 2015-06-17 11:08:40,497 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564515139, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:08:40,536 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=78, retries=350, started=27521 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:40,537 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=219, retries=350, started=56969 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:40,700 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:40,700 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51117: callId: 1538 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:40,704 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:40,704 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51117: callId: 1540 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:40,708 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:40,708 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51117: callId: 1542 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:40,714 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:40,714 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51117: callId: 1544 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:40,725 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:40,725 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51117: callId: 1546 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:40,727 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(197): executing master kill 2015-06-17 11:08:40,727 INFO [MasterRestartThread] hbase.MiniHBaseCluster(422): Aborting 10.0.0.110,51117,1434564506788 2015-06-17 11:08:40,727 FATAL [MasterRestartThread] master.HMaster(1953): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2015-06-17 11:08:40,727 FATAL [MasterRestartThread] master.HMaster(1956): Aborting for tests java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:423) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:270) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$5.run(TestMasterRestartWithProcedures.java:198) 2015-06-17 11:08:40,727 INFO [MasterRestartThread] regionserver.HRegionServer(1859): STOPPED: Aborting for tests 2015-06-17 11:08:40,727 INFO [MasterRestartThread] hbase.LocalHBaseCluster(379): Waiting on 10.0.0.110,51117,1434564506788 2015-06-17 11:08:40,727 INFO [M:0;10.0.0.110:51117] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2015-06-17 11:08:40,727 INFO [M:0;10.0.0.110:51117] regionserver.HeapMemoryManager(201): Stoping HeapMemoryTuner chore. 2015-06-17 11:08:40,727 INFO [SplitLogWorker-10.0.0.110:51117] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2015-06-17 11:08:40,727 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2015-06-17 11:08:40,728 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2015-06-17 11:08:40,727 INFO [M:0;10.0.0.110:51117] procedure2.ProcedureExecutor(490): Stopping the procedure executor 2015-06-17 11:08:40,727 INFO [SplitLogWorker-10.0.0.110:51117] regionserver.SplitLogWorker(155): SplitLogWorker 10.0.0.110,51117,1434564506788 exiting 2015-06-17 11:08:40,728 INFO [M:0;10.0.0.110:51117] wal.WALProcedureStore(159): Stopping the WAL Procedure Store 2015-06-17 11:08:40,733 INFO [IPC Server handler 6 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741881_1057{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 60 2015-06-17 11:08:40,737 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=220, retries=350, started=57169 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:40,737 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=79, retries=350, started=27722 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:40,749 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:40,750 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51117: callId: 1548 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:40,794 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:40,794 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51117: callId: 1550 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:40,822 INFO [10.0.0.110,51117,1434564506788_splitLogManager__ChoreService_1] hbase.ScheduledChore(179): Chore: SplitLogManager Timeout Monitor was stopped 2015-06-17 11:08:40,874 INFO [10.0.0.110,51117,1434564506788_ChoreService_1] hbase.ScheduledChore(179): Chore: 10.0.0.110,51117,1434564506788-MemstoreFlusherChore was stopped 2015-06-17 11:08:40,880 INFO [master//10.0.0.110:0.logRoller] regionserver.LogRoller(159): LogRoller exiting. 2015-06-17 11:08:40,880 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(146): master//10.0.0.110:0.leaseChecker closing leases 2015-06-17 11:08:40,880 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(149): master//10.0.0.110:0.leaseChecker closed leases 2015-06-17 11:08:40,899 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:40,899 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51117: callId: 1552 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:40,937 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=80, retries=350, started=27922 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:40,939 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=221, retries=350, started=57371 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:41,003 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:41,004 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51117: callId: 1554 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:41,107 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:41,107 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51117: callId: 1556 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:41,135 INFO [M:0;10.0.0.110:51117] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2015-06-17 11:08:41,136 INFO [M:0;10.0.0.110:51117] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2015-06-17 11:08:41,136 INFO [M:0;10.0.0.110:51117] regionserver.HRegionServer(1021): stopping server 10.0.0.110,51117,1434564506788 2015-06-17 11:08:41,136 DEBUG [M:0;10.0.0.110:51117] zookeeper.MetaTableLocator(617): Stopping MetaTableLocator 2015-06-17 11:08:41,136 INFO [M:0;10.0.0.110:51117] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320028 2015-06-17 11:08:41,137 DEBUG [M:0;10.0.0.110:51117] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:08:41,138 INFO [M:0;10.0.0.110:51117] regionserver.HRegionServer(1049): stopping server 10.0.0.110,51117,1434564506788; all regions closed. 2015-06-17 11:08:41,138 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=81, retries=350, started=28123 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:41,138 DEBUG [AsyncRpcChannel-pool2-t11] ipc.AsyncRpcChannel$8(574): IPC Client (1076386226) connection to /10.0.0.110:51074 from ndimiduk.hfs.7: closed 2015-06-17 11:08:41,138 DEBUG [M:0;10.0.0.110:51117] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51117,1434564506788 2015-06-17 11:08:41,138 INFO [10.0.0.110:51117.activeMasterManager] zookeeper.MetaTableLocator(340): Failed verification of hbase:meta,,1 at address=10.0.0.110,51074,1434564492618, exception=Call id=0 on server /10.0.0.110:51074 aborted: connection is closing 2015-06-17 11:08:41,139 DEBUG [10.0.0.110:51117.activeMasterManager] zookeeper.ZKUtil(745): master:51117-0x14e02b3a2320027, quorum=localhost:51848, baseZNode=/hbase Unable to get data of znode /hbase/recovering-regions/1588230740 because node does not exist (not an error) 2015-06-17 11:08:41,140 INFO [IPC Server handler 2 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741880_1056{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 91 2015-06-17 11:08:41,141 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=222, retries=350, started=57573 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:41,141 DEBUG [10.0.0.110:51117.activeMasterManager] coordination.ZKSplitLogManagerCoordination(650): Marked 1588230740 recovering from 10.0.0.110,51074,1434564492618: /hbase/recovering-regions/1588230740/10.0.0.110,51074,1434564492618 2015-06-17 11:08:41,141 INFO [10.0.0.110:51117.activeMasterManager] master.HMaster(884): Re-assigning hbase:meta with replicaId, 0 it was on 10.0.0.110,51074,1434564492618 2015-06-17 11:08:41,141 INFO [10.0.0.110:51117.activeMasterManager] master.RegionStates(1072): Transition {1588230740 state=OPEN, ts=1434564506917, server=10.0.0.110,51074,1434564492618} to {1588230740 state=OFFLINE, ts=1434564521141, server=10.0.0.110,51074,1434564492618} 2015-06-17 11:08:41,142 INFO [10.0.0.110:51117.activeMasterManager] zookeeper.MetaTableLocator(440): Setting hbase:meta region location in ZooKeeper as 10.0.0.110,51074,1434564492618 2015-06-17 11:08:41,142 INFO [10.0.0.110:51117.activeMasterManager] master.AssignmentManager(996): Skip assigning hbase:meta,,1.1588230740, the server is stopped/aborted 2015-06-17 11:08:41,142 FATAL [10.0.0.110:51117.activeMasterManager] master.HMaster$1(1546): Failed to become active master java.io.IOException: connection is closed at org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:254) at org.apache.hadoop.hbase.MetaTableAccessor.putToMetaTable(MetaTableAccessor.java:1341) at org.apache.hadoop.hbase.MetaTableAccessor.updateTableState(MetaTableAccessor.java:1691) at org.apache.hadoop.hbase.MetaTableAccessor.updateTableState(MetaTableAccessor.java:1110) at org.apache.hadoop.hbase.master.TableStateManager.udpateMetaState(TableStateManager.java:180) at org.apache.hadoop.hbase.master.TableStateManager.setTableState(TableStateManager.java:67) at org.apache.hadoop.hbase.master.HMaster.assignMeta(HMaster.java:892) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:711) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1542) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:41,211 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:41,212 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51117: callId: 1558 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:41,212 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=513 ms ago, cancelled=false, msg= 2015-06-17 11:08:41,338 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=82, retries=350, started=28323 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:41,341 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=223, retries=350, started=57773 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:41,417 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:41,417 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51117] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51117: callId: 1560 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51126 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:41,417 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=718 ms ago, cancelled=false, msg= 2015-06-17 11:08:41,540 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=83, retries=350, started=28525 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:41,542 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=224, retries=350, started=57974 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:41,546 DEBUG [M:0;10.0.0.110:51117] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:08:41,546 INFO [M:0;10.0.0.110:51117] wal.FSHLog(977): Closed WAL: FSHLog 10.0.0.110%2C51117%2C1434564506788.default:(num 1434564506865) 2015-06-17 11:08:41,546 DEBUG [M:0;10.0.0.110:51117] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:08:41,546 INFO [M:0;10.0.0.110:51117] regionserver.Leases(146): M:0;10.0.0.110:51117 closing leases 2015-06-17 11:08:41,546 INFO [M:0;10.0.0.110:51117] regionserver.Leases(149): M:0;10.0.0.110:51117 closed leases 2015-06-17 11:08:41,547 INFO [M:0;10.0.0.110:51117] hbase.ChoreService(302): Chore service for: 10.0.0.110,51117,1434564506788 had [[ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region 10.0.0.110,51117,1434564506788 Period: 120000 Unit: MILLISECONDS]] on shutdown 2015-06-17 11:08:41,547 INFO [M:0;10.0.0.110:51117] regionserver.CompactSplitThread(394): Waiting for Split Thread to finish... 2015-06-17 11:08:41,547 INFO [M:0;10.0.0.110:51117] regionserver.CompactSplitThread(394): Waiting for Merge Thread to finish... 2015-06-17 11:08:41,547 INFO [M:0;10.0.0.110:51117] regionserver.CompactSplitThread(394): Waiting for Large Compaction Thread to finish... 2015-06-17 11:08:41,547 INFO [M:0;10.0.0.110:51117] regionserver.CompactSplitThread(394): Waiting for Small Compaction Thread to finish... 2015-06-17 11:08:41,549 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51117-0x14e02b3a2320027, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/10.0.0.110,51117,1434564506788 2015-06-17 11:08:41,549 INFO [M:0;10.0.0.110:51117] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a232002b 2015-06-17 11:08:41,550 DEBUG [M:0;10.0.0.110:51117] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:08:41,550 DEBUG [M:0;10.0.0.110:51117] master.HMaster(1074): Stopping service threads 2015-06-17 11:08:41,551 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:08:41,551 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51117-0x14e02b3a2320027, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:08:41,551 INFO [M:0;10.0.0.110:51117] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320029 2015-06-17 11:08:41,552 DEBUG [main-EventThread] zookeeper.ZKUtil(488): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:08:41,552 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(488): master:51117-0x14e02b3a2320027, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:08:41,552 DEBUG [M:0;10.0.0.110:51117] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:08:41,552 INFO [M:0;10.0.0.110:51117] hbase.ChoreService(302): Chore service for: 10.0.0.110,51117,1434564506788_splitLogManager_ had [] on shutdown 2015-06-17 11:08:41,552 INFO [M:0;10.0.0.110:51117] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2015-06-17 11:08:41,552 INFO [M:0;10.0.0.110:51117] ipc.RpcServer(2219): Stopping server on 51117 2015-06-17 11:08:41,552 INFO [RpcServer.listener,port=51117] ipc.RpcServer$Listener(776): RpcServer.listener,port=51117: stopping 2015-06-17 11:08:41,553 INFO [RpcServer.responder] ipc.RpcServer$Responder(1018): RpcServer.responder: stopped 2015-06-17 11:08:41,553 INFO [RpcServer.responder] ipc.RpcServer$Responder(921): RpcServer.responder: stopping 2015-06-17 11:08:41,554 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51117-0x14e02b3a2320027, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,51117,1434564506788 2015-06-17 11:08:41,554 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,51117,1434564506788 2015-06-17 11:08:41,554 INFO [MasterRestartThread-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [10.0.0.110,51117,1434564506788] 2015-06-17 11:08:41,554 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51117-0x14e02b3a2320027, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:08:41,555 INFO [M:0;10.0.0.110:51117] regionserver.HRegionServer(1093): stopping server 10.0.0.110,51117,1434564506788; zookeeper connection closed. 2015-06-17 11:08:41,555 INFO [M:0;10.0.0.110:51117] regionserver.HRegionServer(1096): M:0;10.0.0.110:51117 exiting 2015-06-17 11:08:41,555 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(200): executing master restart 2015-06-17 11:08:41,555 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:41,555 INFO [main-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(121): /hbase/rs/10.0.0.110,51117,1434564506788 znode expired, triggering replicatorRemoved event 2015-06-17 11:08:41,555 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:08:41,556 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:41,567 INFO [ReplicationExecutor-0] replication.ReplicationQueuesZKImpl(171): Atomically moving 10.0.0.110,50957,1434564414812's WALs to my queue 2015-06-17 11:08:41,568 INFO [MasterRestartThread] client.ConnectionUtils(105): master//10.0.0.110:0 server-side HConnection retries=350 2015-06-17 11:08:41,569 INFO [MasterRestartThread] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2015-06-17 11:08:41,570 INFO [MasterRestartThread] ipc.RpcServer$Listener(598): master//10.0.0.110:0: started 10 reader(s). 2015-06-17 11:08:41,572 INFO [MasterRestartThread] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:08:41,573 INFO [MasterRestartThread] zookeeper.RecoverableZooKeeper(121): Process identifier=master:51130 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:08:41,575 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:511300x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:08:41,576 DEBUG [MasterRestartThread] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-17 11:08:41,577 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(578): master:51130-0x14e02b3a232002c connected 2015-06-17 11:08:41,579 DEBUG [MasterRestartThread] zookeeper.ZKUtil(488): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:08:41,579 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2015-06-17 11:08:41,580 INFO [RpcServer.responder] ipc.RpcServer$Responder(917): RpcServer.responder: starting 2015-06-17 11:08:41,580 INFO [RpcServer.listener,port=51130] ipc.RpcServer$Listener(729): RpcServer.listener,port=51130: starting 2015-06-17 11:08:41,580 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=0 queue=0 2015-06-17 11:08:41,580 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=1 queue=0 2015-06-17 11:08:41,580 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=2 queue=0 2015-06-17 11:08:41,580 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=3 queue=0 2015-06-17 11:08:41,580 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=4 queue=0 2015-06-17 11:08:41,580 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=0 queue=0 2015-06-17 11:08:41,580 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=1 queue=1 2015-06-17 11:08:41,580 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=2 queue=0 2015-06-17 11:08:41,581 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=3 queue=1 2015-06-17 11:08:41,581 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=4 queue=0 2015-06-17 11:08:41,581 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=5 queue=1 2015-06-17 11:08:41,581 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=6 queue=0 2015-06-17 11:08:41,581 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=7 queue=1 2015-06-17 11:08:41,581 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=8 queue=0 2015-06-17 11:08:41,581 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=9 queue=1 2015-06-17 11:08:41,581 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=10 queue=0 2015-06-17 11:08:41,581 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=11 queue=1 2015-06-17 11:08:41,582 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=12 queue=0 2015-06-17 11:08:41,582 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=13 queue=1 2015-06-17 11:08:41,582 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=14 queue=0 2015-06-17 11:08:41,582 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=15 queue=1 2015-06-17 11:08:41,582 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=16 queue=0 2015-06-17 11:08:41,582 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=17 queue=1 2015-06-17 11:08:41,582 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=18 queue=0 2015-06-17 11:08:41,582 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=19 queue=1 2015-06-17 11:08:41,582 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=0 queue=0 2015-06-17 11:08:41,583 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=1 queue=0 2015-06-17 11:08:41,583 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=2 queue=0 2015-06-17 11:08:41,583 INFO [MasterRestartThread] master.HMaster(349): hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034, hbase.cluster.distributed=false 2015-06-17 11:08:41,583 INFO [MasterRestartThread] master.HMaster(1512): Adding backup master ZNode /hbase/backup-masters/10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,584 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,585 INFO [M:0;10.0.0.110:51130] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x33ccece9 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:08:41,585 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:08:41,585 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:08:41,586 DEBUG [10.0.0.110:51130.activeMasterManager] zookeeper.ZKUtil(486): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:08:41,586 INFO [10.0.0.110:51130.activeMasterManager] master.ActiveMasterManager(170): Deleting ZNode for /hbase/backup-masters/10.0.0.110,51130,1434564521571 from backup master directory 2015-06-17 11:08:41,587 DEBUG [M:0;10.0.0.110:51130-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x33ccece90x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:08:41,587 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(486): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:08:41,587 DEBUG [MasterRestartThread-EventThread] master.ActiveMasterManager(126): A master is now available 2015-06-17 11:08:41,587 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,587 WARN [10.0.0.110:51130.activeMasterManager] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:08:41,587 INFO [10.0.0.110:51130.activeMasterManager] master.ActiveMasterManager(179): Registered Active Master=10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,588 DEBUG [M:0;10.0.0.110:51130] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2e84fa40, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:08:41,588 DEBUG [M:0;10.0.0.110:51130] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:08:41,588 DEBUG [M:0;10.0.0.110:51130] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:08:41,588 DEBUG [M:0;10.0.0.110:51130-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x33ccece9-0x14e02b3a232002d connected 2015-06-17 11:08:41,589 INFO [M:0;10.0.0.110:51130] regionserver.HRegionServer(774): ClusterId : b7c4b21b-aab2-44b4-8562-2b61af22ad6c 2015-06-17 11:08:41,589 DEBUG [M:0;10.0.0.110:51130] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2015-06-17 11:08:41,590 DEBUG [M:0;10.0.0.110:51130] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:08:41,590 DEBUG [M:0;10.0.0.110:51130] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2015-06-17 11:08:41,590 DEBUG [M:0;10.0.0.110:51130] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2015-06-17 11:08:41,591 DEBUG [M:0;10.0.0.110:51130] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:08:41,591 DEBUG [M:0;10.0.0.110:51130] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2015-06-17 11:08:41,592 INFO [M:0;10.0.0.110:51130] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2015-06-17 11:08:41,592 INFO [M:0;10.0.0.110:51130] compactions.PressureAwareCompactionThroughputController(175): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2015-06-17 11:08:41,592 INFO [M:0;10.0.0.110:51130] regionserver.HRegionServer$CompactionChecker(1504): CompactionChecker runs every 1sec 2015-06-17 11:08:41,592 DEBUG [M:0;10.0.0.110:51130] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@558d480d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=10.0.0.110/10.0.0.110:0 2015-06-17 11:08:41,592 DEBUG [M:0;10.0.0.110:51130] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:08:41,592 DEBUG [M:0;10.0.0.110:51130] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:08:41,592 DEBUG [M:0;10.0.0.110:51130] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:M:0;10.0.0.110:51130 2015-06-17 11:08:41,593 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:08:41,593 DEBUG [M:0;10.0.0.110:51130] zookeeper.ZKUtil(486): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,593 INFO [M:0;10.0.0.110:51130] regionserver.RegionServerCoprocessorHost(65): System coprocessor loading is enabled 2015-06-17 11:08:41,593 INFO [M:0;10.0.0.110:51130] regionserver.RegionServerCoprocessorHost(66): Table coprocessor loading is enabled 2015-06-17 11:08:41,593 INFO [M:0;10.0.0.110:51130] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,51130,1434564521571 with port=51130, startcode=1434564521571 2015-06-17 11:08:41,594 DEBUG [M:0;10.0.0.110:51130] regionserver.HRegionServer(2295): Master is not running yet 2015-06-17 11:08:41,594 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:41,594 WARN [M:0;10.0.0.110:51130] regionserver.HRegionServer(900): reportForDuty failed; sleeping and then retrying. 2015-06-17 11:08:41,594 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,597 DEBUG [10.0.0.110:51130.activeMasterManager] util.FSTableDescriptors(807): Current tableInfoPath = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2015-06-17 11:08:41,599 DEBUG [10.0.0.110:51130.activeMasterManager] util.FSTableDescriptors(811): TableInfo already exists.. Skipping creation 2015-06-17 11:08:41,601 INFO [10.0.0.110:51130.activeMasterManager] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:08:41,601 DEBUG [10.0.0.110:51130.activeMasterManager] coordination.ZKSplitLogManagerCoordination(885): Distributed log replay=true 2015-06-17 11:08:41,602 INFO [10.0.0.110:51130.activeMasterManager] coordination.ZKSplitLogManagerCoordination(599): Found 0 orphan tasks and 0 rescan nodes 2015-06-17 11:08:41,602 DEBUG [10.0.0.110:51130.activeMasterManager] util.FSTableDescriptors(222): Fetching table descriptors from the filesystem. 2015-06-17 11:08:41,610 INFO [10.0.0.110:51130.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x24f8cb4a connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:08:41,612 DEBUG [10.0.0.110:51130.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x24f8cb4a0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:08:41,612 DEBUG [10.0.0.110:51130.activeMasterManager] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45f8fa43, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:08:41,613 DEBUG [10.0.0.110:51130.activeMasterManager] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:08:41,613 DEBUG [10.0.0.110:51130.activeMasterManager] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:08:41,613 DEBUG [10.0.0.110:51130.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x24f8cb4a-0x14e02b3a232002e connected 2015-06-17 11:08:41,614 INFO [10.0.0.110:51130.activeMasterManager] balancer.StochasticLoadBalancer(135): loading config 2015-06-17 11:08:41,614 DEBUG [10.0.0.110:51130.activeMasterManager] zookeeper.ZKUtil(488): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2015-06-17 11:08:41,614 DEBUG [10.0.0.110:51130.activeMasterManager] zookeeper.ZKUtil(486): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:41,615 DEBUG [10.0.0.110:51130.activeMasterManager] zookeeper.ZKUtil(486): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,615 DEBUG [10.0.0.110:51130.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:41,615 DEBUG [10.0.0.110:51130.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,615 INFO [10.0.0.110:51130.activeMasterManager] master.HMaster(568): Server active/primary master=10.0.0.110,51130,1434564521571, sessionid=0x14e02b3a232002c, setting cluster-up flag (Was=true) 2015-06-17 11:08:41,617 DEBUG [10.0.0.110:51130.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:08:41,617 INFO [10.0.0.110:51130.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2015-06-17 11:08:41,617 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:08:41,618 DEBUG [10.0.0.110:51130.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,619 DEBUG [AsyncRpcChannel-pool2-t14] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service MasterService, sasl=false 2015-06-17 11:08:41,619 DEBUG [10.0.0.110:51130.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:08:41,619 DEBUG [RpcServer.listener,port=51130] ipc.RpcServer$Listener(840): RpcServer.listener,port=51130: connection from 10.0.0.110:51136; # active connections: 1 2015-06-17 11:08:41,619 INFO [RpcServer.reader=1,bindAddress=10.0.0.110,port=51130] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51136 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:08:41,619 INFO [10.0.0.110:51130.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2015-06-17 11:08:41,620 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1562 service: MasterService methodName: IsMasterRunning size: 24 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:41,620 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=921 ms ago, cancelled=false, msg= 2015-06-17 11:08:41,620 DEBUG [10.0.0.110:51130.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,620 INFO [10.0.0.110:51130.activeMasterManager] master.MasterCoprocessorHost(86): System coprocessor loading is enabled 2015-06-17 11:08:41,620 DEBUG [10.0.0.110:51130.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_OPEN_REGION-10.0.0.110:51130, corePoolSize=5, maxPoolSize=5 2015-06-17 11:08:41,620 DEBUG [10.0.0.110:51130.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_CLOSE_REGION-10.0.0.110:51130, corePoolSize=5, maxPoolSize=5 2015-06-17 11:08:41,620 DEBUG [10.0.0.110:51130.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_SERVER_OPERATIONS-10.0.0.110:51130, corePoolSize=5, maxPoolSize=5 2015-06-17 11:08:41,621 DEBUG [10.0.0.110:51130.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_META_SERVER_OPERATIONS-10.0.0.110:51130, corePoolSize=5, maxPoolSize=5 2015-06-17 11:08:41,621 DEBUG [10.0.0.110:51130.activeMasterManager] executor.ExecutorService(91): Starting executor service name=M_LOG_REPLAY_OPS-10.0.0.110:51130, corePoolSize=10, maxPoolSize=10 2015-06-17 11:08:41,621 DEBUG [10.0.0.110:51130.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_TABLE_OPERATIONS-10.0.0.110:51130, corePoolSize=1, maxPoolSize=1 2015-06-17 11:08:41,621 INFO [10.0.0.110:51130.activeMasterManager] procedure2.ProcedureExecutor(446): Starting procedure executor threads=9 2015-06-17 11:08:41,621 INFO [10.0.0.110:51130.activeMasterManager] wal.WALProcedureStore(214): Starting WAL Procedure Store lease recovery 2015-06-17 11:08:41,622 INFO [10.0.0.110:51130.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log 2015-06-17 11:08:41,622 INFO [10.0.0.110:51130.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log after 0ms 2015-06-17 11:08:41,622 DEBUG [10.0.0.110:51130.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log; isDirectory=false; length=14915; replication=1; blocksize=134217728; modification_time=1434564413352; access_time=1434564405122; owner=ndimiduk; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:08:41,624 INFO [10.0.0.110:51130.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000008.log 2015-06-17 11:08:41,624 INFO [10.0.0.110:51130.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000008.log after 0ms 2015-06-17 11:08:41,624 DEBUG [10.0.0.110:51130.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000008.log; isDirectory=false; length=60; replication=1; blocksize=134217728; modification_time=1434564521135; access_time=1434564506848; owner=ndimiduk.hfs.7; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:08:41,628 DEBUG [10.0.0.110:51130.activeMasterManager] wal.WALProcedureStore(610): Roll new state log: 9 2015-06-17 11:08:41,629 INFO [10.0.0.110:51130.activeMasterManager] wal.WALProcedureStore(237): Lease acquired for flushLogId: 9 2015-06-17 11:08:41,629 INFO [10.0.0.110:51130.activeMasterManager] wal.ProcedureWALFormatReader(150): No active entry found in state log hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000008.log. removing it 2015-06-17 11:08:41,630 DEBUG [10.0.0.110:51130.activeMasterManager] procedure2.ProcedureExecutor$1(278): load procedures maxProcId=7 2015-06-17 11:08:41,632 DEBUG [10.0.0.110:51130.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=RUNNABLE isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=7 state=RUNNABLE 2015-06-17 11:08:41,632 DEBUG [10.0.0.110:51130.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DeleteTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=6 state=FINISHED 2015-06-17 11:08:41,632 DEBUG [10.0.0.110:51130.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:41,632 DEBUG [10.0.0.110:51130.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=5 state=FINISHED 2015-06-17 11:08:41,632 DEBUG [10.0.0.110:51130.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:41,632 DEBUG [10.0.0.110:51130.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: EnableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=4 state=FINISHED 2015-06-17 11:08:41,632 DEBUG [10.0.0.110:51130.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:41,632 DEBUG [10.0.0.110:51130.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=3 state=FINISHED 2015-06-17 11:08:41,632 DEBUG [10.0.0.110:51130.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:41,632 DEBUG [10.0.0.110:51130.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=2 state=FINISHED 2015-06-17 11:08:41,633 DEBUG [10.0.0.110:51130.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:41,633 DEBUG [10.0.0.110:51130.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=hbase:namespace) user=ndimiduk (auth:SIMPLE) id=1 state=FINISHED 2015-06-17 11:08:41,633 DEBUG [10.0.0.110:51130.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:41,633 DEBUG [10.0.0.110:51130.activeMasterManager] wal.WALProcedureStore(652): Remove log: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000008.log 2015-06-17 11:08:41,634 INFO [IPC Server handler 4 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741881_1057 127.0.0.1:50896 2015-06-17 11:08:41,635 DEBUG [10.0.0.110:51130.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2015-06-17 11:08:41,636 INFO [10.0.0.110:51130.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:08:41,638 DEBUG [10.0.0.110:51130.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:08:41,638 DEBUG [10.0.0.110:51130.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/replication/rs already exists 2015-06-17 11:08:41,639 DEBUG [10.0.0.110:51130.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2015-06-17 11:08:41,639 DEBUG [10.0.0.110:51130.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner 2015-06-17 11:08:41,639 DEBUG [10.0.0.110:51130.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): replicationLogCleaner-0x14e02b3a232002f connected 2015-06-17 11:08:41,639 DEBUG [10.0.0.110:51130.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2015-06-17 11:08:41,640 DEBUG [10.0.0.110:51130.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2015-06-17 11:08:41,640 DEBUG [10.0.0.110:51130.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2015-06-17 11:08:41,640 INFO [10.0.0.110:51130.activeMasterManager] master.ServerManager(1011): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 1, timeout of 4500 ms, interval of 1500 ms. 2015-06-17 11:08:41,640 INFO [M:0;10.0.0.110:51130] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,51130,1434564521571 with port=51130, startcode=1434564521571 2015-06-17 11:08:41,640 INFO [M:0;10.0.0.110:51130] master.ServerManager(459): Registering server=10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,640 INFO [M:0;10.0.0.110:51130] regionserver.HRegionServer(1341): Config from master: hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034 2015-06-17 11:08:41,640 INFO [M:0;10.0.0.110:51130] regionserver.HRegionServer(1341): Config from master: fs.defaultFS=hdfs://localhost:50895 2015-06-17 11:08:41,640 INFO [M:0;10.0.0.110:51130] regionserver.HRegionServer(1341): Config from master: hbase.master.info.port=-1 2015-06-17 11:08:41,640 WARN [M:0;10.0.0.110:51130] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:08:41,640 INFO [M:0;10.0.0.110:51130] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:08:41,640 DEBUG [M:0;10.0.0.110:51130] regionserver.HRegionServer(1603): logdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,643 DEBUG [M:0;10.0.0.110:51130] regionserver.Replication(142): ReplicationStatisticsThread 300 2015-06-17 11:08:41,643 INFO [M:0;10.0.0.110:51130] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:08:41,646 INFO [M:0;10.0.0.110:51130] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.0.0.110%2C51130%2C1434564521571.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51130,1434564521571, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:08:41,652 INFO [M:0;10.0.0.110:51130] wal.FSHLog(1365): Slow sync cost: 3 ms, current pipeline: [] 2015-06-17 11:08:41,652 INFO [M:0;10.0.0.110:51130] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51130,1434564521571/10.0.0.110%2C51130%2C1434564521571.default.1434564521646 2015-06-17 11:08:41,652 INFO [M:0;10.0.0.110:51130] regionserver.MetricsRegionServerWrapperImpl(113): Computing regionserver metrics every 5000 milliseconds 2015-06-17 11:08:41,653 DEBUG [M:0;10.0.0.110:51130] executor.ExecutorService(91): Starting executor service name=RS_OPEN_REGION-10.0.0.110:51130, corePoolSize=3, maxPoolSize=3 2015-06-17 11:08:41,653 DEBUG [M:0;10.0.0.110:51130] executor.ExecutorService(91): Starting executor service name=RS_OPEN_META-10.0.0.110:51130, corePoolSize=1, maxPoolSize=1 2015-06-17 11:08:41,653 DEBUG [M:0;10.0.0.110:51130] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_REGION-10.0.0.110:51130, corePoolSize=3, maxPoolSize=3 2015-06-17 11:08:41,653 DEBUG [M:0;10.0.0.110:51130] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_META-10.0.0.110:51130, corePoolSize=1, maxPoolSize=1 2015-06-17 11:08:41,653 DEBUG [M:0;10.0.0.110:51130] executor.ExecutorService(91): Starting executor service name=RS_LOG_REPLAY_OPS-10.0.0.110:51130, corePoolSize=2, maxPoolSize=2 2015-06-17 11:08:41,654 DEBUG [M:0;10.0.0.110:51130] executor.ExecutorService(91): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-10.0.0.110:51130, corePoolSize=3, maxPoolSize=3 2015-06-17 11:08:41,655 DEBUG [M:0;10.0.0.110:51130] zookeeper.ZKUtil(486): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:41,655 DEBUG [M:0;10.0.0.110:51130] zookeeper.ZKUtil(486): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,655 INFO [M:0;10.0.0.110:51130] regionserver.ReplicationSourceManager(229): Current list of replicators: [10.0.0.110,50905,1434564404764, 10.0.0.110,51130,1434564521571] other RSs: [10.0.0.110,50905,1434564404764, 10.0.0.110,51130,1434564521571] 2015-06-17 11:08:41,667 INFO [M:0;10.0.0.110:51130] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x2e34c442 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:08:41,670 DEBUG [M:0;10.0.0.110:51130-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x2e34c4420x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:08:41,671 DEBUG [M:0;10.0.0.110:51130] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodecWithTags@43eaaa4b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:08:41,671 DEBUG [M:0;10.0.0.110:51130] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:08:41,671 DEBUG [M:0;10.0.0.110:51130] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:08:41,672 DEBUG [M:0;10.0.0.110:51130-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x2e34c442-0x14e02b3a2320030 connected 2015-06-17 11:08:41,683 INFO [M:0;10.0.0.110:51130] regionserver.HeapMemoryManager(190): Starting HeapMemoryTuner chore. 2015-06-17 11:08:41,683 INFO [SplitLogWorker-10.0.0.110:51130] regionserver.SplitLogWorker(134): SplitLogWorker 10.0.0.110,51130,1434564521571 starting 2015-06-17 11:08:41,684 INFO [M:0;10.0.0.110:51130] regionserver.HRegionServer(1363): Serving as 10.0.0.110,51130,1434564521571, RpcServer on 10.0.0.110/10.0.0.110:51130, sessionid=0x14e02b3a232002c 2015-06-17 11:08:41,684 DEBUG [M:0;10.0.0.110:51130] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2015-06-17 11:08:41,684 DEBUG [M:0;10.0.0.110:51130] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager 10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,684 DEBUG [M:0;10.0.0.110:51130] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,51130,1434564521571' 2015-06-17 11:08:41,684 DEBUG [M:0;10.0.0.110:51130] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2015-06-17 11:08:41,684 DEBUG [M:0;10.0.0.110:51130] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2015-06-17 11:08:41,684 DEBUG [M:0;10.0.0.110:51130] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2015-06-17 11:08:41,684 DEBUG [M:0;10.0.0.110:51130] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2015-06-17 11:08:41,684 DEBUG [M:0;10.0.0.110:51130] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager 10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,685 DEBUG [M:0;10.0.0.110:51130] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,51130,1434564521571' 2015-06-17 11:08:41,685 DEBUG [M:0;10.0.0.110:51130] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2015-06-17 11:08:41,685 DEBUG [M:0;10.0.0.110:51130] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2015-06-17 11:08:41,685 DEBUG [M:0;10.0.0.110:51130] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2015-06-17 11:08:41,685 INFO [M:0;10.0.0.110:51130] quotas.RegionServerQuotaManager(62): Quota support disabled 2015-06-17 11:08:41,691 INFO [10.0.0.110:51130.activeMasterManager] master.ServerManager(1028): Finished waiting for region servers count to settle; checked in 1, slept for 51 ms, expecting minimum of 1, maximum of 1, master is running 2015-06-17 11:08:41,691 INFO [10.0.0.110:51130.activeMasterManager] master.ServerManager(459): Registering server=10.0.0.110,50905,1434564404764 2015-06-17 11:08:41,691 INFO [10.0.0.110:51130.activeMasterManager] master.HMaster(665): Registered server found up in zk but who has not yet reported in: 10.0.0.110,50905,1434564404764 2015-06-17 11:08:41,693 INFO [10.0.0.110:51130.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50905,1434564404764 belongs to an existing region server 2015-06-17 11:08:41,696 INFO [10.0.0.110:51130.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51130,1434564521571 belongs to an existing region server 2015-06-17 11:08:41,698 DEBUG [10.0.0.110:51130.activeMasterManager] coordination.ZKSplitLogManagerCoordination(650): Marked 1588230740 recovering from 10.0.0.110,51074,1434564492618: /hbase/recovering-regions/1588230740/10.0.0.110,51074,1434564492618 2015-06-17 11:08:41,698 INFO [10.0.0.110:51130.activeMasterManager] master.HMaster(884): Re-assigning hbase:meta with replicaId, 0 it was on 10.0.0.110,51074,1434564492618 2015-06-17 11:08:41,698 DEBUG [10.0.0.110:51130.activeMasterManager] master.AssignmentManager(1239): No previous transition plan found (or ignoring an existing plan) for hbase:meta,,1.1588230740; generated random plan=hri=hbase:meta,,1.1588230740, src=, dest=10.0.0.110,51130,1434564521571; 2 (online=2) available servers, forceNewPlan=false 2015-06-17 11:08:41,698 INFO [10.0.0.110:51130.activeMasterManager] master.AssignmentManager(1028): Assigning hbase:meta,,1.1588230740 to 10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,698 INFO [10.0.0.110:51130.activeMasterManager] master.RegionStates(1072): Transition {1588230740 state=OFFLINE, ts=1434564521698, server=10.0.0.110,51074,1434564492618} to {1588230740 state=PENDING_OPEN, ts=1434564521698, server=10.0.0.110,51130,1434564521571} 2015-06-17 11:08:41,698 INFO [10.0.0.110:51130.activeMasterManager] zookeeper.MetaTableLocator(440): Setting hbase:meta region location in ZooKeeper as 10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,699 DEBUG [10.0.0.110:51130.activeMasterManager] master.ServerManager(937): New admin connection to 10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,699 INFO [10.0.0.110:51130.activeMasterManager] regionserver.RSRpcServices(1432): Open hbase:meta,,1.1588230740 2015-06-17 11:08:41,700 DEBUG [10.0.0.110:51130.activeMasterManager] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564521700,"tag":[],"qualifier":"state","vlen":2}]},"row":"hbase:meta"} 2015-06-17 11:08:41,700 INFO [RS_OPEN_META-10.0.0.110:51130-0] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:08:41,702 INFO [RS_OPEN_META-10.0.0.110:51130-0] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.0.0.110%2C51130%2C1434564521571..meta, suffix=.meta, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51130,1434564521571, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:08:41,707 INFO [RS_OPEN_META-10.0.0.110:51130-0] wal.FSHLog(1365): Slow sync cost: 3 ms, current pipeline: [] 2015-06-17 11:08:41,707 INFO [RS_OPEN_META-10.0.0.110:51130-0] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51130,1434564521571/10.0.0.110%2C51130%2C1434564521571..meta.1434564521702.meta 2015-06-17 11:08:41,708 DEBUG [RS_OPEN_META-10.0.0.110:51130-0] regionserver.HRegion(6070): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2015-06-17 11:08:41,709 DEBUG [RS_OPEN_META-10.0.0.110:51130-0] coprocessor.CoprocessorHost(179): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2015-06-17 11:08:41,709 DEBUG [RS_OPEN_META-10.0.0.110:51130-0] regionserver.HRegion(7284): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2015-06-17 11:08:41,709 INFO [RS_OPEN_META-10.0.0.110:51130-0] regionserver.RegionCoprocessorHost(350): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2015-06-17 11:08:41,709 DEBUG [RS_OPEN_META-10.0.0.110:51130-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table meta 1588230740 2015-06-17 11:08:41,709 DEBUG [RS_OPEN_META-10.0.0.110:51130-0] regionserver.HRegion(722): Instantiated hbase:meta,,1.1588230740 2015-06-17 11:08:41,711 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:08:41,711 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:08:41,718 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/0cdbef2057bf48ffab39a58d41d9feda, isReference=false, isBulkLoadResult=false, seqid=120000147, majorCompaction=false 2015-06-17 11:08:41,723 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/b97cb9bad56b4097bc8516a91cd2d5c4, isReference=false, isBulkLoadResult=false, seqid=80000131, majorCompaction=true 2015-06-17 11:08:41,723 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:08:41,724 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:08:41,730 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/309e63f83f3a4850bf003589102a7393, isReference=false, isBulkLoadResult=false, seqid=80000131, majorCompaction=true 2015-06-17 11:08:41,736 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/9810d8ec635a40f199322a209fb0a8f6, isReference=false, isBulkLoadResult=false, seqid=120000147, majorCompaction=false 2015-06-17 11:08:41,738 DEBUG [RS_OPEN_META-10.0.0.110:51130-0] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740 2015-06-17 11:08:41,739 DEBUG [RS_OPEN_META-10.0.0.110:51130-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:08:41,740 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=84, retries=350, started=28725 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:41,742 DEBUG [RS_OPEN_META-10.0.0.110:51130-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/recovered.edits/160000150.seqid to file, newSeqId=160000150, maxSeqId=120000150 2015-06-17 11:08:41,743 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=225, retries=350, started=58175 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:41,743 INFO [RS_OPEN_META-10.0.0.110:51130-0] regionserver.HRegion(848): Onlined 1588230740; next sequenceid=160000150 2015-06-17 11:08:41,743 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1893): Post open deploy tasks for hbase:meta,,1.1588230740 2015-06-17 11:08:41,744 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/recovering-regions/1588230740 2015-06-17 11:08:41,745 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(3181): Update last flushed sequence id of region 1588230740 for 10.0.0.110,51074,1434564492618 2015-06-17 11:08:41,745 DEBUG [PostOpenDeployTasks:1588230740] master.AssignmentManager(2812): Got transition OPENED for {1588230740 state=PENDING_OPEN, ts=1434564521698, server=10.0.0.110,51130,1434564521571} from 10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,745 INFO [PostOpenDeployTasks:1588230740] master.RegionStates(1072): Transition {1588230740 state=PENDING_OPEN, ts=1434564521698, server=10.0.0.110,51130,1434564521571} to {1588230740 state=OPEN, ts=1434564521745, server=10.0.0.110,51130,1434564521571} 2015-06-17 11:08:41,745 INFO [PostOpenDeployTasks:1588230740] zookeeper.MetaTableLocator(440): Setting hbase:meta region location in ZooKeeper as 10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,746 DEBUG [PostOpenDeployTasks:1588230740] master.RegionStates(417): Onlined 1588230740 on 10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,746 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1920): Finished post open deploy task for hbase:meta,,1.1588230740 2015-06-17 11:08:41,746 DEBUG [RS_OPEN_META-10.0.0.110:51130-0] handler.OpenRegionHandler(125): Opened hbase:meta,,1.1588230740 on 10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,821 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:08:41,823 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:41,823 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1565 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:41,824 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1125 ms ago, cancelled=false, msg= 2015-06-17 11:08:41,903 DEBUG [AsyncRpcChannel-pool2-t15] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service ClientService, sasl=false 2015-06-17 11:08:41,903 DEBUG [RpcServer.listener,port=51130] ipc.RpcServer$Listener(840): RpcServer.listener,port=51130: connection from 10.0.0.110:51144; # active connections: 2 2015-06-17 11:08:41,904 INFO [RpcServer.reader=2,bindAddress=10.0.0.110,port=51130] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51144 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:08:41,906 INFO [10.0.0.110:51130.activeMasterManager] hbase.MetaTableAccessor(1692): Updated table hbase:meta state to ENABLED in META 2015-06-17 11:08:41,907 DEBUG [10.0.0.110:51130.activeMasterManager] master.MasterFileSystem(339): Renamed region directory: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51074,1434564492618-splitting 2015-06-17 11:08:41,907 INFO [10.0.0.110:51130.activeMasterManager] master.SplitLogManager(560): dead splitlog workers [10.0.0.110,51074,1434564492618] 2015-06-17 11:08:41,908 INFO [10.0.0.110:51130.activeMasterManager] master.SplitLogManager(192): hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51074,1434564492618-splitting is empty dir, no logs to split 2015-06-17 11:08:41,908 DEBUG [10.0.0.110:51130.activeMasterManager] master.SplitLogManager(255): Scheduling batch of logs to split 2015-06-17 11:08:41,908 INFO [10.0.0.110:51130.activeMasterManager] master.SplitLogManager(257): started splitting 0 logs in [hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51074,1434564492618-splitting] for [10.0.0.110,51074,1434564492618] 2015-06-17 11:08:41,909 DEBUG [10.0.0.110:51130.activeMasterManager] coordination.ZKSplitLogManagerCoordination(324): Processing recovering [1588230740] and servers [10.0.0.110,51074,1434564492618], isMetaRecovery=true 2015-06-17 11:08:41,911 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/recovering-regions/1588230740/10.0.0.110,51074,1434564492618 2015-06-17 11:08:41,911 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/recovering-regions/1588230740 2015-06-17 11:08:41,912 INFO [10.0.0.110:51130.activeMasterManager] master.SplitLogManager(316): finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51074,1434564492618-splitting] in 4ms 2015-06-17 11:08:41,912 INFO [RS_OPEN_REGION-10.0.0.110:51130-0] handler.FinishRegionRecoveringHandler(52): /hbase/recovering-regions/1588230740 deleted; 1588230740 recovered. 2015-06-17 11:08:41,913 DEBUG [10.0.0.110:51130.activeMasterManager] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564521912,"tag":[],"qualifier":"state","vlen":2}]},"row":"hbase:meta"} 2015-06-17 11:08:41,915 INFO [10.0.0.110:51130.activeMasterManager] hbase.MetaTableAccessor(1692): Updated table hbase:meta state to ENABLED in META 2015-06-17 11:08:41,921 INFO [10.0.0.110:51130.activeMasterManager] master.ServerManager(686): AssignmentManager hasn't finished failover cleanup; waiting 2015-06-17 11:08:41,922 INFO [10.0.0.110:51130.activeMasterManager] master.HMaster(911): hbase:meta with replicaId 0 assigned=1, location=10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,939 INFO [10.0.0.110:51130.activeMasterManager] master.RegionStates(1072): Transition {9ccc7b1e8942a5472c9cf2d844c2fff0 state=OPEN, ts=1434564521935, server=10.0.0.110,51074,1434564492618} to {9ccc7b1e8942a5472c9cf2d844c2fff0 state=CLOSED, ts=1434564521939, server=10.0.0.110,51074,1434564492618} 2015-06-17 11:08:41,939 INFO [10.0.0.110:51130.activeMasterManager] master.AssignmentManager(538): Clean cluster startup. Assigning user regions 2015-06-17 11:08:41,939 INFO [10.0.0.110:51130.activeMasterManager] master.AssignmentManager(680): Assigning 1 region(s) to 10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,940 INFO [10.0.0.110:51130.activeMasterManager] master.RegionStates(1072): Transition {9ccc7b1e8942a5472c9cf2d844c2fff0 state=CLOSED, ts=1434564521939, server=10.0.0.110,51074,1434564492618} to {9ccc7b1e8942a5472c9cf2d844c2fff0 state=PENDING_OPEN, ts=1434564521940, server=10.0.0.110,51130,1434564521571} 2015-06-17 11:08:41,940 INFO [10.0.0.110:51130.activeMasterManager] master.RegionStateStore(207): Updating row hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. with state=PENDING_OPEN&sn=10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,941 INFO [10.0.0.110:51130.activeMasterManager] regionserver.RSRpcServices(1432): Open hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:08:41,941 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=85, retries=350, started=28926 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:41,942 DEBUG [10.0.0.110:51130.activeMasterManager] master.AssignmentManager(851): Bulk assigning done for 10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,942 DEBUG [RS_OPEN_REGION-10.0.0.110:51130-1] regionserver.HRegion(6070): Opening region: {ENCODED => 9ccc7b1e8942a5472c9cf2d844c2fff0, NAME => 'hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.', STARTKEY => '', ENDKEY => ''} 2015-06-17 11:08:41,943 DEBUG [RS_OPEN_REGION-10.0.0.110:51130-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table namespace 9ccc7b1e8942a5472c9cf2d844c2fff0 2015-06-17 11:08:41,943 DEBUG [RS_OPEN_REGION-10.0.0.110:51130-1] regionserver.HRegion(722): Instantiated hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:08:41,943 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=226, retries=350, started=58375 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:41,944 INFO [StoreOpener-9ccc7b1e8942a5472c9cf2d844c2fff0-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=4, currentSize=1077384, freeSize=1042884920, maxSize=1043962304, heapSize=1077384, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:08:41,945 INFO [StoreOpener-9ccc7b1e8942a5472c9cf2d844c2fff0-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:08:41,949 INFO [10.0.0.110:51130.activeMasterManager] master.AssignmentManager(420): Joined the cluster in 27ms, failover=false 2015-06-17 11:08:41,953 DEBUG [StoreOpener-9ccc7b1e8942a5472c9cf2d844c2fff0-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/info/b88c94ff75ab4fafba9b040819600c6e, isReference=false, isBulkLoadResult=false, seqid=8, majorCompaction=false 2015-06-17 11:08:41,954 DEBUG [10.0.0.110:51130.activeMasterManager] zookeeper.ZKUtil(486): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2015-06-17 11:08:41,955 DEBUG [10.0.0.110:51130.activeMasterManager] hbase.ZKNamespaceManager(196): Updating namespace cache from node default with data: \x0A\x07default 2015-06-17 11:08:41,955 DEBUG [10.0.0.110:51130.activeMasterManager] hbase.ZKNamespaceManager(196): Updating namespace cache from node hbase with data: \x0A\x05hbase 2015-06-17 11:08:41,955 DEBUG [RS_OPEN_REGION-10.0.0.110:51130-1] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0 2015-06-17 11:08:41,957 DEBUG [RS_OPEN_REGION-10.0.0.110:51130-1] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:08:41,960 DEBUG [RS_OPEN_REGION-10.0.0.110:51130-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/recovered.edits/21.seqid to file, newSeqId=21, maxSeqId=20 2015-06-17 11:08:41,961 INFO [RS_OPEN_REGION-10.0.0.110:51130-1] regionserver.HRegion(848): Onlined 9ccc7b1e8942a5472c9cf2d844c2fff0; next sequenceid=21 2015-06-17 11:08:41,962 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] regionserver.HRegionServer(1893): Post open deploy tasks for hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:08:41,962 DEBUG [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.AssignmentManager(2812): Got transition OPENED for {9ccc7b1e8942a5472c9cf2d844c2fff0 state=PENDING_OPEN, ts=1434564521940, server=10.0.0.110,51130,1434564521571} from 10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,962 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStates(1072): Transition {9ccc7b1e8942a5472c9cf2d844c2fff0 state=PENDING_OPEN, ts=1434564521940, server=10.0.0.110,51130,1434564521571} to {9ccc7b1e8942a5472c9cf2d844c2fff0 state=OPEN, ts=1434564521962, server=10.0.0.110,51130,1434564521571} 2015-06-17 11:08:41,962 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStateStore(207): Updating row hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. with state=OPEN&openSeqNum=21&server=10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,963 DEBUG [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStates(417): Onlined 9ccc7b1e8942a5472c9cf2d844c2fff0 on 10.0.0.110,51130,1434564521571 2015-06-17 11:08:41,963 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStates(429): Offlined 9ccc7b1e8942a5472c9cf2d844c2fff0 from 10.0.0.110,51074,1434564492618 2015-06-17 11:08:41,964 DEBUG [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] regionserver.HRegionServer(1920): Finished post open deploy task for hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:08:41,964 DEBUG [RS_OPEN_REGION-10.0.0.110:51130-1] handler.OpenRegionHandler(125): Opened hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. on 10.0.0.110,51130,1434564521571 2015-06-17 11:08:42,025 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:42,025 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1567 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:42,026 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1327 ms ago, cancelled=false, msg= 2015-06-17 11:08:42,142 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=86, retries=350, started=29127 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:42,143 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=227, retries=350, started=58575 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:42,227 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:42,227 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51130: callId: 1569 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:42,228 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1529 ms ago, cancelled=false, msg= 2015-06-17 11:08:42,342 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=87, retries=350, started=29327 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:42,344 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=228, retries=350, started=58776 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:42,430 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:42,430 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51130: callId: 1571 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:42,430 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1731 ms ago, cancelled=false, msg= 2015-06-17 11:08:42,543 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=88, retries=350, started=29528 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:42,544 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=229, retries=350, started=58976 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:42,631 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:42,631 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1573 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:42,632 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1933 ms ago, cancelled=false, msg= 2015-06-17 11:08:42,745 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=89, retries=350, started=29730 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:42,745 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=230, retries=350, started=59177 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:42,834 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:42,834 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1575 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:42,834 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2135 ms ago, cancelled=false, msg= 2015-06-17 11:08:42,945 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=90, retries=350, started=29930 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:42,947 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=231, retries=350, started=59379 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:43,035 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:43,036 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1577 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:43,036 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2337 ms ago, cancelled=false, msg= 2015-06-17 11:08:43,147 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=91, retries=350, started=30132 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:43,149 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=232, retries=350, started=59581 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:43,237 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:43,237 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51130: callId: 1579 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:43,238 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2539 ms ago, cancelled=false, msg= 2015-06-17 11:08:43,348 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=92, retries=350, started=30333 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:43,349 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=233, retries=350, started=59781 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:43,440 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:43,440 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51130: callId: 1581 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:43,440 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2741 ms ago, cancelled=false, msg= 2015-06-17 11:08:43,535 DEBUG [AsyncRpcChannel-pool2-t1] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2015-06-17 11:08:43,535 DEBUG [RpcServer.listener,port=51130] ipc.RpcServer$Listener(840): RpcServer.listener,port=51130: connection from 10.0.0.110:51147; # active connections: 3 2015-06-17 11:08:43,536 INFO [RpcServer.reader=3,bindAddress=10.0.0.110,port=51130] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51147 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:08:43,549 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=93, retries=350, started=30534 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:43,550 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=234, retries=350, started=59982 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:43,641 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:43,641 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1583 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:43,642 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2943 ms ago, cancelled=false, msg= 2015-06-17 11:08:43,749 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=94, retries=350, started=30734 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:43,751 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=235, retries=350, started=60183 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:43,844 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:43,844 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1585 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:43,845 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3146 ms ago, cancelled=false, msg= 2015-06-17 11:08:43,949 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=95, retries=350, started=30934 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:43,952 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=236, retries=350, started=60384 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:44,046 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:44,046 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1587 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:44,046 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3347 ms ago, cancelled=false, msg= 2015-06-17 11:08:44,151 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=96, retries=350, started=31136 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:44,154 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=237, retries=350, started=60586 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:44,248 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:44,248 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51130: callId: 1589 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:44,249 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3550 ms ago, cancelled=false, msg= 2015-06-17 11:08:44,351 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=97, retries=350, started=31336 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:44,356 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=238, retries=350, started=60788 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:44,450 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:44,451 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51130: callId: 1591 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:44,451 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3752 ms ago, cancelled=false, msg= 2015-06-17 11:08:44,551 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=98, retries=350, started=31536 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:44,557 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=239, retries=350, started=60989 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:44,652 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:44,652 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1593 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:44,652 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3953 ms ago, cancelled=false, msg= 2015-06-17 11:08:44,752 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=99, retries=350, started=31737 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:44,758 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=240, retries=350, started=61190 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:44,855 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:44,856 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1595 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:44,856 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4157 ms ago, cancelled=false, msg= 2015-06-17 11:08:44,954 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=100, retries=350, started=31939 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:44,960 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=241, retries=350, started=61392 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:45,057 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:45,057 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1597 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:45,058 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4359 ms ago, cancelled=false, msg= 2015-06-17 11:08:45,076 DEBUG [10.0.0.110,50905,1434564404764_ChoreService_1] compactions.PressureAwareCompactionThroughputController(148): compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-06-17 11:08:45,154 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=101, retries=350, started=32139 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:45,160 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=242, retries=350, started=61592 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:45,259 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:45,259 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51130: callId: 1599 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:45,259 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4560 ms ago, cancelled=false, msg= 2015-06-17 11:08:45,354 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=102, retries=350, started=32339 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:45,362 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=243, retries=350, started=61794 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:45,462 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:45,462 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51130: callId: 1601 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:45,462 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4763 ms ago, cancelled=false, msg= 2015-06-17 11:08:45,556 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=103, retries=350, started=32541 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:45,564 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=244, retries=350, started=61996 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:45,664 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:45,664 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1603 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:45,664 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4965 ms ago, cancelled=false, msg= 2015-06-17 11:08:45,758 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=104, retries=350, started=32743 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:45,765 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=245, retries=350, started=62197 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:45,865 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:45,865 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1605 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:45,866 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5167 ms ago, cancelled=false, msg= 2015-06-17 11:08:45,958 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=105, retries=350, started=32943 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:45,965 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=246, retries=350, started=62397 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:46,068 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:46,068 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1607 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:46,068 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5369 ms ago, cancelled=false, msg= 2015-06-17 11:08:46,069 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564520699, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:08:46,158 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=106, retries=350, started=33143 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:46,165 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=247, retries=350, started=62597 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:46,272 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:46,272 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51130: callId: 1609 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:46,275 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:46,275 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51130: callId: 1611 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:46,279 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:46,279 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1613 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:46,285 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:46,285 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1615 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:46,296 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:46,296 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1617 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:46,318 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:46,318 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51130: callId: 1619 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:46,359 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=107, retries=350, started=33344 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:46,359 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:46,360 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51130: callId: 1621 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:46,366 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=248, retries=350, started=62798 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:46,462 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:46,462 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1623 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:46,560 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=108, retries=350, started=33545 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:46,563 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:46,563 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1625 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:46,567 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=249, retries=350, started=62999 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:46,665 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:46,665 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1627 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:46,761 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=109, retries=350, started=33746 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:46,766 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:46,766 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51130: callId: 1629 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:46,766 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=495 ms ago, cancelled=false, msg= 2015-06-17 11:08:46,768 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=250, retries=350, started=63200 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:46,962 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=110, retries=350, started=33947 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:46,967 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:46,967 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51130: callId: 1631 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:46,968 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=697 ms ago, cancelled=false, msg= 2015-06-17 11:08:46,969 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=251, retries=350, started=63401 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:47,163 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=111, retries=350, started=34148 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:47,169 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:47,169 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1633 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:47,169 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=898 ms ago, cancelled=false, msg= 2015-06-17 11:08:47,169 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=252, retries=350, started=63601 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:47,364 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=112, retries=350, started=34349 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:47,370 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=253, retries=350, started=63802 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:47,371 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:47,371 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1635 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:47,372 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1101 ms ago, cancelled=false, msg= 2015-06-17 11:08:47,564 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=113, retries=350, started=34549 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:47,572 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=254, retries=350, started=64004 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:47,572 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:47,573 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1637 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:47,573 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1302 ms ago, cancelled=false, msg= 2015-06-17 11:08:47,766 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=114, retries=350, started=34751 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:47,772 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=255, retries=350, started=64204 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:47,774 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:47,774 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51130: callId: 1639 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:47,774 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1503 ms ago, cancelled=false, msg= 2015-06-17 11:08:47,967 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=115, retries=350, started=34952 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:47,972 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=256, retries=350, started=64404 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:47,976 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:47,976 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51130: callId: 1641 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:47,977 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1706 ms ago, cancelled=false, msg= 2015-06-17 11:08:48,168 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=116, retries=350, started=35153 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:48,173 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=257, retries=350, started=64605 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:48,178 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:48,178 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1643 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:48,178 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1907 ms ago, cancelled=false, msg= 2015-06-17 11:08:48,369 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=117, retries=350, started=35354 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:48,374 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=258, retries=350, started=64806 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:48,381 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:48,381 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1645 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:48,382 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2111 ms ago, cancelled=false, msg= 2015-06-17 11:08:48,570 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=118, retries=350, started=35555 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:48,575 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=259, retries=350, started=65007 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:48,583 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:48,583 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1647 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:48,583 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2312 ms ago, cancelled=false, msg= 2015-06-17 11:08:48,772 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=119, retries=350, started=35757 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:48,776 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=260, retries=350, started=65208 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:48,785 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:48,785 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51130: callId: 1649 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:48,786 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2515 ms ago, cancelled=false, msg= 2015-06-17 11:08:48,972 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=120, retries=350, started=35957 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:48,977 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=261, retries=350, started=65409 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:48,987 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:48,987 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51130: callId: 1651 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:48,987 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2716 ms ago, cancelled=false, msg= 2015-06-17 11:08:49,174 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=121, retries=350, started=36159 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:49,177 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=262, retries=350, started=65609 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:49,189 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:49,189 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1653 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:49,189 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2918 ms ago, cancelled=false, msg= 2015-06-17 11:08:49,375 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=122, retries=350, started=36360 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:49,378 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=263, retries=350, started=65810 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:49,391 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:49,391 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1655 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:49,391 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3120 ms ago, cancelled=false, msg= 2015-06-17 11:08:49,577 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=123, retries=350, started=36562 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:49,579 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=264, retries=350, started=66011 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:49,593 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:49,593 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1657 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:49,594 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3323 ms ago, cancelled=false, msg= 2015-06-17 11:08:49,779 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=124, retries=350, started=36764 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:49,779 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=265, retries=350, started=66211 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:49,796 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:49,796 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51130: callId: 1659 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:49,797 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3526 ms ago, cancelled=false, msg= 2015-06-17 11:08:49,979 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=266, retries=350, started=66411 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:49,980 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=125, retries=350, started=36965 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:49,999 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:49,999 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51130: callId: 1661 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:49,999 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3728 ms ago, cancelled=false, msg= 2015-06-17 11:08:50,179 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=267, retries=350, started=66611 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:50,180 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=126, retries=350, started=37165 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:50,201 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:50,201 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1663 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:50,201 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3930 ms ago, cancelled=false, msg= 2015-06-17 11:08:50,381 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=268, retries=350, started=66813 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:50,382 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=127, retries=350, started=37367 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:50,403 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:50,403 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1665 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:50,403 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4132 ms ago, cancelled=false, msg= 2015-06-17 11:08:50,582 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=128, retries=350, started=37567 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:50,582 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=269, retries=350, started=67014 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:50,604 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:50,604 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1667 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:50,604 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4333 ms ago, cancelled=false, msg= 2015-06-17 11:08:50,782 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=129, retries=350, started=37767 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:50,783 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=270, retries=350, started=67215 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:50,807 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:50,807 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51130: callId: 1669 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:50,807 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4536 ms ago, cancelled=false, msg= 2015-06-17 11:08:50,984 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=271, retries=350, started=67416 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:50,984 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=130, retries=350, started=37969 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:51,009 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:51,009 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51130: callId: 1671 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:51,009 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4738 ms ago, cancelled=false, msg= 2015-06-17 11:08:51,184 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=272, retries=350, started=67616 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:51,186 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=131, retries=350, started=38171 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:51,212 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:51,213 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1673 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:51,213 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4942 ms ago, cancelled=false, msg= 2015-06-17 11:08:51,386 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=273, retries=350, started=67818 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:51,386 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=132, retries=350, started=38371 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:51,416 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:51,416 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1675 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:51,416 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5145 ms ago, cancelled=false, msg= 2015-06-17 11:08:51,586 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=274, retries=350, started=68018 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:51,586 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=133, retries=350, started=38571 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:51,618 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:51,619 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1677 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:51,619 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5348 ms ago, cancelled=false, msg= 2015-06-17 11:08:51,619 DEBUG [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(188): Sleeping for 6616 before executing next master restart count=8 2015-06-17 11:08:51,620 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564526271, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:08:51,789 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=275, retries=350, started=68221 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:51,789 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=134, retries=350, started=38774 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:51,821 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:51,822 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51130: callId: 1679 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:51,824 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:51,824 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51130: callId: 1681 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:51,829 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:51,829 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1683 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:51,835 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:51,835 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1685 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:51,846 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:51,846 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1687 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:51,867 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:51,867 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51130: callId: 1689 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:51,909 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:51,909 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51130: callId: 1691 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:51,989 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=276, retries=350, started=68421 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:51,989 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=135, retries=350, started=38974 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:52,011 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:52,011 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1693 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:52,112 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:52,112 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1695 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:52,189 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=136, retries=350, started=39174 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:52,191 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=277, retries=350, started=68623 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:52,214 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:52,215 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1697 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:52,316 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:52,316 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51130: callId: 1699 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:52,316 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=495 ms ago, cancelled=false, msg= 2015-06-17 11:08:52,390 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=137, retries=350, started=39375 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:52,392 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=278, retries=350, started=68824 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:52,519 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:52,519 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51130: callId: 1701 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:52,520 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=699 ms ago, cancelled=false, msg= 2015-06-17 11:08:52,591 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=138, retries=350, started=39575 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:52,592 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=279, retries=350, started=69024 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:52,721 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:52,721 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1703 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:52,721 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=900 ms ago, cancelled=false, msg= 2015-06-17 11:08:52,792 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=139, retries=350, started=39777 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:52,793 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=280, retries=350, started=69225 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:52,922 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:52,923 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1705 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:52,923 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1102 ms ago, cancelled=false, msg= 2015-06-17 11:08:52,992 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=140, retries=350, started=39977 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:52,994 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=281, retries=350, started=69426 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:53,125 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:53,125 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1707 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:53,125 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1304 ms ago, cancelled=false, msg= 2015-06-17 11:08:53,193 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=141, retries=350, started=40178 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:53,195 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=282, retries=350, started=69627 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:53,327 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:53,327 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51130: callId: 1709 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:53,327 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1506 ms ago, cancelled=false, msg= 2015-06-17 11:08:53,394 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=142, retries=350, started=40379 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:53,397 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=283, retries=350, started=69828 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:53,530 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:53,531 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51130: callId: 1711 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:53,531 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1710 ms ago, cancelled=false, msg= 2015-06-17 11:08:53,595 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=143, retries=350, started=40580 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:53,599 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=284, retries=350, started=70031 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:53,732 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:53,732 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1713 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:53,732 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1911 ms ago, cancelled=false, msg= 2015-06-17 11:08:53,796 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=144, retries=350, started=40781 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:53,800 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=285, retries=350, started=70232 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:53,934 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:53,934 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1715 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:53,935 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2114 ms ago, cancelled=false, msg= 2015-06-17 11:08:53,998 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=145, retries=350, started=40983 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:54,001 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=286, retries=350, started=70433 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:54,136 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:54,136 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1717 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:54,137 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2316 ms ago, cancelled=false, msg= 2015-06-17 11:08:54,199 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=146, retries=350, started=41184 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:54,202 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=287, retries=350, started=70634 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:54,339 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:54,339 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51130: callId: 1719 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:54,340 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2519 ms ago, cancelled=false, msg= 2015-06-17 11:08:54,400 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=147, retries=350, started=41385 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:54,403 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=288, retries=350, started=70835 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:54,542 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:54,542 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51130: callId: 1721 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:54,542 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2721 ms ago, cancelled=false, msg= 2015-06-17 11:08:54,600 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=148, retries=350, started=41585 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:54,604 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=289, retries=350, started=71036 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:54,744 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:54,744 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1723 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:54,745 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2924 ms ago, cancelled=false, msg= 2015-06-17 11:08:54,802 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=149, retries=350, started=41787 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:54,805 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=290, retries=350, started=71237 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:54,947 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:54,947 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1725 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:54,947 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3126 ms ago, cancelled=false, msg= 2015-06-17 11:08:55,002 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=150, retries=350, started=41987 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:55,006 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=291, retries=350, started=71438 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:55,150 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:55,150 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1727 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:55,150 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3329 ms ago, cancelled=false, msg= 2015-06-17 11:08:55,204 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=151, retries=350, started=42189 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:55,208 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=292, retries=350, started=71640 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:55,353 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:55,353 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51130: callId: 1729 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:55,354 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3533 ms ago, cancelled=false, msg= 2015-06-17 11:08:55,405 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=152, retries=350, started=42390 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:55,409 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=293, retries=350, started=71841 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:55,556 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:55,556 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51130: callId: 1731 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:55,557 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3736 ms ago, cancelled=false, msg= 2015-06-17 11:08:55,607 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=153, retries=350, started=42592 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:55,610 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=294, retries=350, started=72042 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:55,758 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:55,758 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1733 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:55,758 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3937 ms ago, cancelled=false, msg= 2015-06-17 11:08:55,808 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=154, retries=350, started=42793 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:55,811 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=295, retries=350, started=72243 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:55,960 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:55,961 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1735 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:55,961 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4140 ms ago, cancelled=false, msg= 2015-06-17 11:08:56,009 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=155, retries=350, started=42994 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:56,012 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=296, retries=350, started=72444 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:56,163 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:56,163 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1737 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:56,163 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4342 ms ago, cancelled=false, msg= 2015-06-17 11:08:56,211 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=156, retries=350, started=43196 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:56,212 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=297, retries=350, started=72644 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:56,365 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:56,365 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51130: callId: 1739 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:56,366 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4545 ms ago, cancelled=false, msg= 2015-06-17 11:08:56,411 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=157, retries=350, started=43396 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:56,413 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=298, retries=350, started=72845 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:56,567 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:56,567 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51130: callId: 1741 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:56,567 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4746 ms ago, cancelled=false, msg= 2015-06-17 11:08:56,612 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=158, retries=350, started=43597 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:56,615 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=299, retries=350, started=73047 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:56,769 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:56,769 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1743 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:56,769 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4948 ms ago, cancelled=false, msg= 2015-06-17 11:08:56,814 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=159, retries=350, started=43799 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:56,816 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=300, retries=350, started=73248 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:56,970 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:56,971 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1745 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:56,971 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5150 ms ago, cancelled=false, msg= 2015-06-17 11:08:57,015 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=160, retries=350, started=44000 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:57,017 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=301, retries=350, started=73449 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:57,173 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:57,173 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1747 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:57,173 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5352 ms ago, cancelled=false, msg= 2015-06-17 11:08:57,174 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564531821, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:08:57,217 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=161, retries=350, started=44202 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:57,220 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=302, retries=350, started=73652 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:57,376 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:57,376 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51130: callId: 1749 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:57,379 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:57,379 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51130: callId: 1751 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:57,383 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:57,383 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1753 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:57,389 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:57,389 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1755 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:57,400 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:57,400 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1757 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:57,418 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=162, retries=350, started=44403 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:57,421 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=303, retries=350, started=73853 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:57,421 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:57,421 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51130: callId: 1759 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:57,463 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:57,463 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51130: callId: 1761 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:57,565 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:57,565 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1763 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:57,618 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=163, retries=350, started=44603 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:57,621 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=304, retries=350, started=74053 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:57,666 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:57,667 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1765 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:57,768 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:57,768 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1767 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:57,819 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=164, retries=350, started=44804 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:57,823 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=305, retries=350, started=74255 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:57,870 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:57,870 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51130: callId: 1769 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:57,871 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=496 ms ago, cancelled=false, msg= 2015-06-17 11:08:58,019 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=165, retries=350, started=45004 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:58,023 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=306, retries=350, started=74455 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:58,073 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:58,073 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51130: callId: 1771 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:58,073 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=698 ms ago, cancelled=false, msg= 2015-06-17 11:08:58,219 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=166, retries=350, started=45204 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:58,224 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=307, retries=350, started=74656 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:58,236 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(197): executing master kill 2015-06-17 11:08:58,237 INFO [MasterRestartThread] hbase.MiniHBaseCluster(422): Aborting 10.0.0.110,51130,1434564521571 2015-06-17 11:08:58,237 FATAL [MasterRestartThread] master.HMaster(1953): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2015-06-17 11:08:58,237 FATAL [MasterRestartThread] master.HMaster(1956): Aborting for tests java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:423) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:270) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$5.run(TestMasterRestartWithProcedures.java:198) 2015-06-17 11:08:58,237 INFO [MasterRestartThread] regionserver.HRegionServer(1859): STOPPED: Aborting for tests 2015-06-17 11:08:58,237 INFO [MasterRestartThread] hbase.LocalHBaseCluster(379): Waiting on 10.0.0.110,51130,1434564521571 2015-06-17 11:08:58,237 INFO [M:0;10.0.0.110:51130] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2015-06-17 11:08:58,237 INFO [M:0;10.0.0.110:51130] regionserver.HeapMemoryManager(201): Stoping HeapMemoryTuner chore. 2015-06-17 11:08:58,237 INFO [SplitLogWorker-10.0.0.110:51130] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2015-06-17 11:08:58,237 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2015-06-17 11:08:58,238 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2015-06-17 11:08:58,237 INFO [M:0;10.0.0.110:51130] procedure2.ProcedureExecutor(490): Stopping the procedure executor 2015-06-17 11:08:58,237 INFO [SplitLogWorker-10.0.0.110:51130] regionserver.SplitLogWorker(155): SplitLogWorker 10.0.0.110,51130,1434564521571 exiting 2015-06-17 11:08:58,238 INFO [M:0;10.0.0.110:51130] wal.WALProcedureStore(159): Stopping the WAL Procedure Store 2015-06-17 11:08:58,245 INFO [IPC Server handler 0 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741884_1060{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 0 2015-06-17 11:08:58,245 INFO [M:0;10.0.0.110:51130] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2015-06-17 11:08:58,245 INFO [M:0;10.0.0.110:51130] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2015-06-17 11:08:58,246 INFO [M:0;10.0.0.110:51130] regionserver.HRegionServer(1021): stopping server 10.0.0.110,51130,1434564521571 2015-06-17 11:08:58,246 DEBUG [M:0;10.0.0.110:51130] zookeeper.MetaTableLocator(617): Stopping MetaTableLocator 2015-06-17 11:08:58,246 DEBUG [RS_CLOSE_REGION-10.0.0.110:51130-0] handler.CloseRegionHandler(90): Processing close of hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:08:58,246 DEBUG [RS_CLOSE_REGION-10.0.0.110:51130-0] regionserver.HRegion(1365): Closing hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.: disabling compactions & flushes 2015-06-17 11:08:58,246 DEBUG [RS_CLOSE_REGION-10.0.0.110:51130-0] regionserver.HRegion(1392): Updates disabled for region hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:08:58,247 INFO [M:0;10.0.0.110:51130] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a232002d 2015-06-17 11:08:58,247 INFO [StoreCloserThread-hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.-1] regionserver.HStore(870): Closed info 2015-06-17 11:08:58,247 DEBUG [M:0;10.0.0.110:51130] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:08:58,248 INFO [M:0;10.0.0.110:51130] regionserver.CompactSplitThread(394): Waiting for Split Thread to finish... 2015-06-17 11:08:58,248 DEBUG [AsyncRpcChannel-pool2-t15] ipc.AsyncRpcChannel$8(574): IPC Client (544718118) connection to /10.0.0.110:51130 from ndimiduk.hfs.8: closed 2015-06-17 11:08:58,248 DEBUG [RpcServer.reader=2,bindAddress=10.0.0.110,port=51130] ipc.RpcServer$Listener(872): RpcServer.listener,port=51130: DISCONNECTING client 10.0.0.110:51144 because read count=-1. Number of active connections: 3 2015-06-17 11:08:58,248 INFO [M:0;10.0.0.110:51130] regionserver.CompactSplitThread(394): Waiting for Merge Thread to finish... 2015-06-17 11:08:58,248 INFO [M:0;10.0.0.110:51130] regionserver.CompactSplitThread(394): Waiting for Large Compaction Thread to finish... 2015-06-17 11:08:58,248 INFO [M:0;10.0.0.110:51130] regionserver.CompactSplitThread(394): Waiting for Small Compaction Thread to finish... 2015-06-17 11:08:58,248 DEBUG [AsyncRpcChannel-pool2-t16] ipc.AsyncRpcChannel$8(574): IPC Client (1965823302) connection to /10.0.0.110:51074 from ndimiduk.hfs.8: closed 2015-06-17 11:08:58,249 INFO [M:0;10.0.0.110:51130] regionserver.HRegionServer(1243): Waiting on 2 regions to close 2015-06-17 11:08:58,249 DEBUG [M:0;10.0.0.110:51130] regionserver.HRegionServer(1247): {9ccc7b1e8942a5472c9cf2d844c2fff0=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., 1588230740=hbase:meta,,1.1588230740} 2015-06-17 11:08:58,249 DEBUG [RS_CLOSE_META-10.0.0.110:51130-0] handler.CloseRegionHandler(90): Processing close of hbase:meta,,1.1588230740 2015-06-17 11:08:58,249 DEBUG [RS_CLOSE_META-10.0.0.110:51130-0] regionserver.HRegion(1365): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2015-06-17 11:08:58,249 DEBUG [RS_CLOSE_META-10.0.0.110:51130-0] regionserver.HRegion(1392): Updates disabled for region hbase:meta,,1.1588230740 2015-06-17 11:08:58,249 INFO [RS_CLOSE_META-10.0.0.110:51130-0] regionserver.HRegion(2111): Flushing 2/2 column families, memstore=1.75 KB 2015-06-17 11:08:58,252 DEBUG [RS_CLOSE_REGION-10.0.0.110:51130-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/recovered.edits/23.seqid to file, newSeqId=23, maxSeqId=21 2015-06-17 11:08:58,253 INFO [RS_CLOSE_REGION-10.0.0.110:51130-0] regionserver.HRegion(1486): Closed hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:08:58,253 DEBUG [RS_CLOSE_REGION-10.0.0.110:51130-0] handler.CloseRegionHandler(122): Closed hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:08:58,257 INFO [IPC Server handler 5 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741885_1061{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 0 2015-06-17 11:08:58,257 INFO [RS_CLOSE_META-10.0.0.110:51130-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=160000161, memsize=1.4 K, hasBloomFilter=false, into tmp file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/ebcc205b8b154961b65112b14f4a05ba 2015-06-17 11:08:58,282 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:58,282 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51130: callId: 1773 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:58,282 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=907 ms ago, cancelled=false, msg= 2015-06-17 11:08:58,288 INFO [IPC Server handler 6 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741886_1062{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 0 2015-06-17 11:08:58,289 INFO [RS_CLOSE_META-10.0.0.110:51130-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=160000161, memsize=352, hasBloomFilter=false, into tmp file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/1b0c1561002046f7ac7b7f7dcc004d33 2015-06-17 11:08:58,295 DEBUG [RS_CLOSE_META-10.0.0.110:51130-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/ebcc205b8b154961b65112b14f4a05ba as hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/ebcc205b8b154961b65112b14f4a05ba 2015-06-17 11:08:58,301 INFO [RS_CLOSE_META-10.0.0.110:51130-0] regionserver.HStore(963): Added hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/ebcc205b8b154961b65112b14f4a05ba, entries=6, sequenceid=160000161, filesize=5.3 K 2015-06-17 11:08:58,302 DEBUG [RS_CLOSE_META-10.0.0.110:51130-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/1b0c1561002046f7ac7b7f7dcc004d33 as hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/1b0c1561002046f7ac7b7f7dcc004d33 2015-06-17 11:08:58,308 INFO [RS_CLOSE_META-10.0.0.110:51130-0] regionserver.HStore(963): Added hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/1b0c1561002046f7ac7b7f7dcc004d33, entries=2, sequenceid=160000161, filesize=4.6 K 2015-06-17 11:08:58,309 INFO [RS_CLOSE_META-10.0.0.110:51130-0] regionserver.HRegion(2400): Finished memstore flush of ~1.75 KB/1792, currentsize=0 B/0 for region hbase:meta,,1.1588230740 in 60ms, sequenceid=160000161, compaction requested=true 2015-06-17 11:08:58,310 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(870): Closed info 2015-06-17 11:08:58,311 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(870): Closed table 2015-06-17 11:08:58,315 DEBUG [RS_CLOSE_META-10.0.0.110:51130-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/recovered.edits/160000164.seqid to file, newSeqId=160000164, maxSeqId=160000150 2015-06-17 11:08:58,316 DEBUG [RS_CLOSE_META-10.0.0.110:51130-0] coprocessor.CoprocessorHost(269): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2015-06-17 11:08:58,316 INFO [RS_CLOSE_META-10.0.0.110:51130-0] regionserver.HRegion(1486): Closed hbase:meta,,1.1588230740 2015-06-17 11:08:58,316 DEBUG [RS_CLOSE_META-10.0.0.110:51130-0] handler.CloseRegionHandler(122): Closed hbase:meta,,1.1588230740 2015-06-17 11:08:58,422 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=167, retries=350, started=45406 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:58,425 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=308, retries=350, started=74857 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:58,449 INFO [M:0;10.0.0.110:51130] regionserver.HRegionServer(1049): stopping server 10.0.0.110,51130,1434564521571; all regions closed. 2015-06-17 11:08:58,449 DEBUG [M:0;10.0.0.110:51130] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51130,1434564521571 2015-06-17 11:08:58,453 INFO [IPC Server handler 6 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741883_1059{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 1934 2015-06-17 11:08:58,488 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:58,488 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51130: callId: 1775 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:58,488 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1113 ms ago, cancelled=false, msg= 2015-06-17 11:08:58,602 INFO [10.0.0.110,51130,1434564521571_splitLogManager__ChoreService_1] hbase.ScheduledChore(179): Chore: SplitLogManager Timeout Monitor was stopped 2015-06-17 11:08:58,624 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=168, retries=350, started=45609 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:58,626 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=309, retries=350, started=75058 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:58,654 INFO [10.0.0.110,51130,1434564521571_ChoreService_1] hbase.ScheduledChore(179): Chore: 10.0.0.110,51130,1434564521571-MemstoreFlusherChore was stopped 2015-06-17 11:08:58,660 INFO [master//10.0.0.110:0.logRoller] regionserver.LogRoller(159): LogRoller exiting. 2015-06-17 11:08:58,660 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(146): master//10.0.0.110:0.leaseChecker closing leases 2015-06-17 11:08:58,661 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(149): master//10.0.0.110:0.leaseChecker closed leases 2015-06-17 11:08:58,692 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:58,692 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51130] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51130: callId: 1777 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51136 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:58,693 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1318 ms ago, cancelled=false, msg= 2015-06-17 11:08:58,711 INFO [RS_OPEN_META-10.0.0.110:51130-0-MetaLogRoller] regionserver.LogRoller(159): LogRoller exiting. 2015-06-17 11:08:58,825 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=169, retries=350, started=45810 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:58,828 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=310, retries=350, started=75259 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:58,858 DEBUG [M:0;10.0.0.110:51130] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:08:58,858 INFO [M:0;10.0.0.110:51130] wal.FSHLog(977): Closed WAL: FSHLog 10.0.0.110%2C51130%2C1434564521571..meta:.meta(num 1434564521702) 2015-06-17 11:08:58,858 DEBUG [M:0;10.0.0.110:51130] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51130,1434564521571 2015-06-17 11:08:58,862 INFO [IPC Server handler 3 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741882_1058{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 83 2015-06-17 11:08:58,865 DEBUG [M:0;10.0.0.110:51130] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:08:58,865 INFO [M:0;10.0.0.110:51130] wal.FSHLog(977): Closed WAL: FSHLog 10.0.0.110%2C51130%2C1434564521571.default:(num 1434564521646) 2015-06-17 11:08:58,865 DEBUG [M:0;10.0.0.110:51130] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:08:58,865 INFO [M:0;10.0.0.110:51130] regionserver.Leases(146): M:0;10.0.0.110:51130 closing leases 2015-06-17 11:08:58,865 INFO [M:0;10.0.0.110:51130] regionserver.Leases(149): M:0;10.0.0.110:51130 closed leases 2015-06-17 11:08:58,866 INFO [M:0;10.0.0.110:51130] hbase.ChoreService(302): Chore service for: 10.0.0.110,51130,1434564521571 had [[ScheduledChore: Name: 10.0.0.110,51130,1434564521571-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-10.0.0.110:51130 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.0.0.110,51130,1434564521571-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region 10.0.0.110,51130,1434564521571 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS]] on shutdown 2015-06-17 11:08:58,868 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/10.0.0.110,51130,1434564521571 2015-06-17 11:08:58,868 INFO [M:0;10.0.0.110:51130] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320030 2015-06-17 11:08:58,869 DEBUG [M:0;10.0.0.110:51130] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:08:58,869 DEBUG [M:0;10.0.0.110:51130] master.HMaster(1074): Stopping service threads 2015-06-17 11:08:58,870 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:08:58,870 INFO [M:0;10.0.0.110:51130] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a232002e 2015-06-17 11:08:58,870 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:08:58,870 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(488): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:08:58,871 DEBUG [M:0;10.0.0.110:51130] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:08:58,871 DEBUG [main-EventThread] zookeeper.ZKUtil(488): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:08:58,871 INFO [M:0;10.0.0.110:51130] hbase.ChoreService(302): Chore service for: 10.0.0.110,51130,1434564521571_splitLogManager_ had [] on shutdown 2015-06-17 11:08:58,871 INFO [M:0;10.0.0.110:51130] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2015-06-17 11:08:58,871 INFO [M:0;10.0.0.110:51130] ipc.RpcServer(2219): Stopping server on 51130 2015-06-17 11:08:58,871 INFO [RpcServer.listener,port=51130] ipc.RpcServer$Listener(776): RpcServer.listener,port=51130: stopping 2015-06-17 11:08:58,872 INFO [RpcServer.responder] ipc.RpcServer$Responder(1018): RpcServer.responder: stopped 2015-06-17 11:08:58,872 INFO [RpcServer.responder] ipc.RpcServer$Responder(921): RpcServer.responder: stopping 2015-06-17 11:08:58,873 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,51130,1434564521571 2015-06-17 11:08:58,873 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,51130,1434564521571 2015-06-17 11:08:58,873 INFO [MasterRestartThread-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [10.0.0.110,51130,1434564521571] 2015-06-17 11:08:58,873 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51130-0x14e02b3a232002c, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:08:58,873 INFO [M:0;10.0.0.110:51130] regionserver.HRegionServer(1093): stopping server 10.0.0.110,51130,1434564521571; zookeeper connection closed. 2015-06-17 11:08:58,874 INFO [M:0;10.0.0.110:51130] regionserver.HRegionServer(1096): M:0;10.0.0.110:51130 exiting 2015-06-17 11:08:58,874 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(200): executing master restart 2015-06-17 11:08:58,874 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:58,874 INFO [main-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(121): /hbase/rs/10.0.0.110,51130,1434564521571 znode expired, triggering replicatorRemoved event 2015-06-17 11:08:58,874 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:08:58,874 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:58,890 INFO [MasterRestartThread] client.ConnectionUtils(105): master//10.0.0.110:0 server-side HConnection retries=350 2015-06-17 11:08:58,890 INFO [MasterRestartThread] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2015-06-17 11:08:58,894 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:08:58,895 DEBUG [Thread-199] zookeeper.ZKUtil(745): hconnection-0x51e5f22a-0x14e02b3a2320008, quorum=localhost:51848, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2015-06-17 11:08:58,896 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1521 ms ago, cancelled=false, msg= 2015-06-17 11:08:58,918 INFO [MasterRestartThread] ipc.RpcServer$Listener(598): master//10.0.0.110:0: started 10 reader(s). 2015-06-17 11:08:58,921 INFO [MasterRestartThread] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:08:58,922 INFO [MasterRestartThread] zookeeper.RecoverableZooKeeper(121): Process identifier=master:51155 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:08:58,924 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:511550x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:08:58,925 DEBUG [MasterRestartThread] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-17 11:08:58,925 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(578): master:51155-0x14e02b3a2320031 connected 2015-06-17 11:08:58,927 DEBUG [MasterRestartThread] zookeeper.ZKUtil(488): master:51155-0x14e02b3a2320031, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:08:58,927 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:51155-0x14e02b3a2320031, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2015-06-17 11:08:58,928 INFO [RpcServer.responder] ipc.RpcServer$Responder(917): RpcServer.responder: starting 2015-06-17 11:08:58,928 INFO [RpcServer.listener,port=51155] ipc.RpcServer$Listener(729): RpcServer.listener,port=51155: starting 2015-06-17 11:08:58,928 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=0 queue=0 2015-06-17 11:08:58,928 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=1 queue=0 2015-06-17 11:08:58,928 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=2 queue=0 2015-06-17 11:08:58,928 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=3 queue=0 2015-06-17 11:08:58,928 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=4 queue=0 2015-06-17 11:08:58,928 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=0 queue=0 2015-06-17 11:08:58,928 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=1 queue=1 2015-06-17 11:08:58,928 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=2 queue=0 2015-06-17 11:08:58,929 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=3 queue=1 2015-06-17 11:08:58,929 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=4 queue=0 2015-06-17 11:08:58,929 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=5 queue=1 2015-06-17 11:08:58,929 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=6 queue=0 2015-06-17 11:08:58,929 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=7 queue=1 2015-06-17 11:08:58,929 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=8 queue=0 2015-06-17 11:08:58,929 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=9 queue=1 2015-06-17 11:08:58,929 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=10 queue=0 2015-06-17 11:08:58,929 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=11 queue=1 2015-06-17 11:08:58,930 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=12 queue=0 2015-06-17 11:08:58,930 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=13 queue=1 2015-06-17 11:08:58,930 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=14 queue=0 2015-06-17 11:08:58,930 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=15 queue=1 2015-06-17 11:08:58,930 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=16 queue=0 2015-06-17 11:08:58,930 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=17 queue=1 2015-06-17 11:08:58,930 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=18 queue=0 2015-06-17 11:08:58,930 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=19 queue=1 2015-06-17 11:08:58,930 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=0 queue=0 2015-06-17 11:08:58,931 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=1 queue=0 2015-06-17 11:08:58,931 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=2 queue=0 2015-06-17 11:08:58,931 INFO [MasterRestartThread] master.HMaster(349): hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034, hbase.cluster.distributed=false 2015-06-17 11:08:58,931 INFO [MasterRestartThread] master.HMaster(1512): Adding backup master ZNode /hbase/backup-masters/10.0.0.110,51155,1434564538919 2015-06-17 11:08:58,932 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:51155-0x14e02b3a2320031, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/10.0.0.110,51155,1434564538919 2015-06-17 11:08:58,933 INFO [M:0;10.0.0.110:51155] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0xeadd8dd connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:08:58,933 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51155-0x14e02b3a2320031, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:08:58,933 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:08:58,933 DEBUG [10.0.0.110:51155.activeMasterManager] zookeeper.ZKUtil(486): master:51155-0x14e02b3a2320031, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:08:58,934 INFO [10.0.0.110:51155.activeMasterManager] master.ActiveMasterManager(170): Deleting ZNode for /hbase/backup-masters/10.0.0.110,51155,1434564538919 from backup master directory 2015-06-17 11:08:58,934 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(486): master:51155-0x14e02b3a2320031, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:08:58,934 DEBUG [MasterRestartThread-EventThread] master.ActiveMasterManager(126): A master is now available 2015-06-17 11:08:58,935 DEBUG [M:0;10.0.0.110:51155-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0xeadd8dd0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:08:58,935 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51155-0x14e02b3a2320031, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/10.0.0.110,51155,1434564538919 2015-06-17 11:08:58,935 WARN [10.0.0.110:51155.activeMasterManager] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:08:58,935 INFO [10.0.0.110:51155.activeMasterManager] master.ActiveMasterManager(179): Registered Active Master=10.0.0.110,51155,1434564538919 2015-06-17 11:08:58,935 DEBUG [M:0;10.0.0.110:51155] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@17447166, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:08:58,936 DEBUG [M:0;10.0.0.110:51155] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:08:58,936 DEBUG [M:0;10.0.0.110:51155] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:08:58,936 DEBUG [M:0;10.0.0.110:51155-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0xeadd8dd-0x14e02b3a2320032 connected 2015-06-17 11:08:58,937 INFO [M:0;10.0.0.110:51155] regionserver.HRegionServer(774): ClusterId : b7c4b21b-aab2-44b4-8562-2b61af22ad6c 2015-06-17 11:08:58,937 DEBUG [M:0;10.0.0.110:51155] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2015-06-17 11:08:58,937 DEBUG [M:0;10.0.0.110:51155] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:08:58,938 DEBUG [M:0;10.0.0.110:51155] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2015-06-17 11:08:58,938 DEBUG [M:0;10.0.0.110:51155] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2015-06-17 11:08:58,939 DEBUG [M:0;10.0.0.110:51155] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:08:58,939 DEBUG [M:0;10.0.0.110:51155] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2015-06-17 11:08:58,939 INFO [M:0;10.0.0.110:51155] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2015-06-17 11:08:58,940 INFO [M:0;10.0.0.110:51155] compactions.PressureAwareCompactionThroughputController(175): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2015-06-17 11:08:58,940 INFO [M:0;10.0.0.110:51155] regionserver.HRegionServer$CompactionChecker(1504): CompactionChecker runs every 1sec 2015-06-17 11:08:58,941 DEBUG [M:0;10.0.0.110:51155] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f1eb085, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=10.0.0.110/10.0.0.110:0 2015-06-17 11:08:58,941 DEBUG [M:0;10.0.0.110:51155] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:08:58,941 DEBUG [M:0;10.0.0.110:51155] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:08:58,941 DEBUG [M:0;10.0.0.110:51155] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:M:0;10.0.0.110:51155 2015-06-17 11:08:58,941 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:08:58,942 DEBUG [M:0;10.0.0.110:51155] zookeeper.ZKUtil(486): master:51155-0x14e02b3a2320031, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51155,1434564538919 2015-06-17 11:08:58,942 INFO [M:0;10.0.0.110:51155] regionserver.RegionServerCoprocessorHost(65): System coprocessor loading is enabled 2015-06-17 11:08:58,942 INFO [M:0;10.0.0.110:51155] regionserver.RegionServerCoprocessorHost(66): Table coprocessor loading is enabled 2015-06-17 11:08:58,942 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:58,942 INFO [M:0;10.0.0.110:51155] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,51155,1434564538919 with port=51155, startcode=1434564538919 2015-06-17 11:08:58,942 DEBUG [M:0;10.0.0.110:51155] regionserver.HRegionServer(2295): Master is not running yet 2015-06-17 11:08:58,942 WARN [M:0;10.0.0.110:51155] regionserver.HRegionServer(900): reportForDuty failed; sleeping and then retrying. 2015-06-17 11:08:58,942 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51155,1434564538919 2015-06-17 11:08:58,943 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=350, started=16988 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:08:58,947 DEBUG [10.0.0.110:51155.activeMasterManager] util.FSTableDescriptors(807): Current tableInfoPath = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2015-06-17 11:08:58,949 DEBUG [10.0.0.110:51155.activeMasterManager] util.FSTableDescriptors(811): TableInfo already exists.. Skipping creation 2015-06-17 11:08:58,952 INFO [10.0.0.110:51155.activeMasterManager] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:08:58,952 DEBUG [10.0.0.110:51155.activeMasterManager] coordination.ZKSplitLogManagerCoordination(885): Distributed log replay=true 2015-06-17 11:08:58,953 INFO [10.0.0.110:51155.activeMasterManager] coordination.ZKSplitLogManagerCoordination(599): Found 0 orphan tasks and 0 rescan nodes 2015-06-17 11:08:58,953 DEBUG [10.0.0.110:51155.activeMasterManager] util.FSTableDescriptors(222): Fetching table descriptors from the filesystem. 2015-06-17 11:08:58,962 INFO [10.0.0.110:51155.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x4b1c44ee connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:08:58,964 DEBUG [10.0.0.110:51155.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x4b1c44ee0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:08:58,965 DEBUG [10.0.0.110:51155.activeMasterManager] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1940fa6a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:08:58,965 DEBUG [10.0.0.110:51155.activeMasterManager] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:08:58,965 DEBUG [10.0.0.110:51155.activeMasterManager] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:08:58,965 DEBUG [10.0.0.110:51155.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x4b1c44ee-0x14e02b3a2320033 connected 2015-06-17 11:08:58,966 INFO [10.0.0.110:51155.activeMasterManager] balancer.StochasticLoadBalancer(135): loading config 2015-06-17 11:08:58,966 DEBUG [10.0.0.110:51155.activeMasterManager] zookeeper.ZKUtil(488): master:51155-0x14e02b3a2320031, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2015-06-17 11:08:58,967 DEBUG [10.0.0.110:51155.activeMasterManager] zookeeper.ZKUtil(486): master:51155-0x14e02b3a2320031, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:58,967 DEBUG [10.0.0.110:51155.activeMasterManager] zookeeper.ZKUtil(486): master:51155-0x14e02b3a2320031, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51155,1434564538919 2015-06-17 11:08:58,967 DEBUG [10.0.0.110:51155.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:58,967 DEBUG [10.0.0.110:51155.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,51155,1434564538919 2015-06-17 11:08:58,968 INFO [10.0.0.110:51155.activeMasterManager] master.HMaster(568): Server active/primary master=10.0.0.110,51155,1434564538919, sessionid=0x14e02b3a2320031, setting cluster-up flag (Was=true) 2015-06-17 11:08:58,969 DEBUG [10.0.0.110:51155.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:08:58,970 INFO [10.0.0.110:51155.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2015-06-17 11:08:58,970 DEBUG [10.0.0.110:51155.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,51155,1434564538919 2015-06-17 11:08:58,971 DEBUG [10.0.0.110:51155.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:08:58,972 INFO [10.0.0.110:51155.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2015-06-17 11:08:58,972 DEBUG [10.0.0.110:51155.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,51155,1434564538919 2015-06-17 11:08:58,972 INFO [10.0.0.110:51155.activeMasterManager] master.MasterCoprocessorHost(86): System coprocessor loading is enabled 2015-06-17 11:08:58,972 DEBUG [10.0.0.110:51155.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_OPEN_REGION-10.0.0.110:51155, corePoolSize=5, maxPoolSize=5 2015-06-17 11:08:58,972 DEBUG [10.0.0.110:51155.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_CLOSE_REGION-10.0.0.110:51155, corePoolSize=5, maxPoolSize=5 2015-06-17 11:08:58,972 DEBUG [10.0.0.110:51155.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_SERVER_OPERATIONS-10.0.0.110:51155, corePoolSize=5, maxPoolSize=5 2015-06-17 11:08:58,973 DEBUG [10.0.0.110:51155.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_META_SERVER_OPERATIONS-10.0.0.110:51155, corePoolSize=5, maxPoolSize=5 2015-06-17 11:08:58,973 DEBUG [10.0.0.110:51155.activeMasterManager] executor.ExecutorService(91): Starting executor service name=M_LOG_REPLAY_OPS-10.0.0.110:51155, corePoolSize=10, maxPoolSize=10 2015-06-17 11:08:58,973 DEBUG [10.0.0.110:51155.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_TABLE_OPERATIONS-10.0.0.110:51155, corePoolSize=1, maxPoolSize=1 2015-06-17 11:08:58,973 INFO [10.0.0.110:51155.activeMasterManager] procedure2.ProcedureExecutor(446): Starting procedure executor threads=9 2015-06-17 11:08:58,973 INFO [10.0.0.110:51155.activeMasterManager] wal.WALProcedureStore(214): Starting WAL Procedure Store lease recovery 2015-06-17 11:08:58,974 INFO [10.0.0.110:51155.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log 2015-06-17 11:08:58,975 INFO [10.0.0.110:51155.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log after 1ms 2015-06-17 11:08:58,975 DEBUG [10.0.0.110:51155.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log; isDirectory=false; length=14915; replication=1; blocksize=134217728; modification_time=1434564413352; access_time=1434564405122; owner=ndimiduk; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:08:58,977 INFO [10.0.0.110:51155.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000009.log 2015-06-17 11:08:58,977 INFO [10.0.0.110:51155.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000009.log after 0ms 2015-06-17 11:08:58,977 DEBUG [10.0.0.110:51155.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000009.log; isDirectory=false; length=60; replication=1; blocksize=134217728; modification_time=1434564538245; access_time=1434564521627; owner=ndimiduk.hfs.8; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:08:58,980 DEBUG [10.0.0.110:51155.activeMasterManager] wal.WALProcedureStore(610): Roll new state log: 10 2015-06-17 11:08:58,981 INFO [10.0.0.110:51155.activeMasterManager] wal.WALProcedureStore(237): Lease acquired for flushLogId: 10 2015-06-17 11:08:58,982 INFO [10.0.0.110:51155.activeMasterManager] wal.ProcedureWALFormatReader(150): No active entry found in state log hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000009.log. removing it 2015-06-17 11:08:58,983 DEBUG [10.0.0.110:51155.activeMasterManager] procedure2.ProcedureExecutor$1(278): load procedures maxProcId=7 2015-06-17 11:08:58,984 DEBUG [10.0.0.110:51155.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=RUNNABLE isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=7 state=RUNNABLE 2015-06-17 11:08:58,984 DEBUG [10.0.0.110:51155.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DeleteTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=6 state=FINISHED 2015-06-17 11:08:58,984 DEBUG [10.0.0.110:51155.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:58,984 DEBUG [10.0.0.110:51155.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=5 state=FINISHED 2015-06-17 11:08:58,985 DEBUG [10.0.0.110:51155.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:58,985 DEBUG [10.0.0.110:51155.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: EnableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=4 state=FINISHED 2015-06-17 11:08:58,985 DEBUG [10.0.0.110:51155.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:58,985 DEBUG [10.0.0.110:51155.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=3 state=FINISHED 2015-06-17 11:08:58,985 DEBUG [10.0.0.110:51155.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:58,985 DEBUG [10.0.0.110:51155.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=2 state=FINISHED 2015-06-17 11:08:58,985 DEBUG [10.0.0.110:51155.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:58,985 DEBUG [10.0.0.110:51155.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=hbase:namespace) user=ndimiduk (auth:SIMPLE) id=1 state=FINISHED 2015-06-17 11:08:58,985 DEBUG [10.0.0.110:51155.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:08:58,985 DEBUG [10.0.0.110:51155.activeMasterManager] wal.WALProcedureStore(652): Remove log: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000009.log 2015-06-17 11:08:58,986 INFO [IPC Server handler 8 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741884_1060 127.0.0.1:50896 2015-06-17 11:08:58,987 DEBUG [10.0.0.110:51155.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2015-06-17 11:08:58,988 INFO [10.0.0.110:51155.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:08:58,989 DEBUG [10.0.0.110:51155.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:08:58,990 DEBUG [10.0.0.110:51155.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/replication/rs already exists 2015-06-17 11:08:58,990 DEBUG [10.0.0.110:51155.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2015-06-17 11:08:58,991 DEBUG [10.0.0.110:51155.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner 2015-06-17 11:08:58,991 DEBUG [10.0.0.110:51155.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): replicationLogCleaner-0x14e02b3a2320034 connected 2015-06-17 11:08:58,992 DEBUG [10.0.0.110:51155.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2015-06-17 11:08:58,992 DEBUG [10.0.0.110:51155.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2015-06-17 11:08:58,992 DEBUG [10.0.0.110:51155.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2015-06-17 11:08:58,992 INFO [10.0.0.110:51155.activeMasterManager] master.ServerManager(1011): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 1, timeout of 4500 ms, interval of 1500 ms. 2015-06-17 11:08:58,992 INFO [M:0;10.0.0.110:51155] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,51155,1434564538919 with port=51155, startcode=1434564538919 2015-06-17 11:08:58,993 INFO [M:0;10.0.0.110:51155] master.ServerManager(459): Registering server=10.0.0.110,51155,1434564538919 2015-06-17 11:08:58,993 INFO [M:0;10.0.0.110:51155] regionserver.HRegionServer(1341): Config from master: hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034 2015-06-17 11:08:58,993 INFO [M:0;10.0.0.110:51155] regionserver.HRegionServer(1341): Config from master: fs.defaultFS=hdfs://localhost:50895 2015-06-17 11:08:58,993 INFO [M:0;10.0.0.110:51155] regionserver.HRegionServer(1341): Config from master: hbase.master.info.port=-1 2015-06-17 11:08:58,993 WARN [M:0;10.0.0.110:51155] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:08:58,993 INFO [M:0;10.0.0.110:51155] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:08:58,993 DEBUG [M:0;10.0.0.110:51155] regionserver.HRegionServer(1603): logdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51155,1434564538919 2015-06-17 11:08:58,995 DEBUG [M:0;10.0.0.110:51155] regionserver.Replication(142): ReplicationStatisticsThread 300 2015-06-17 11:08:58,995 INFO [M:0;10.0.0.110:51155] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:08:58,998 INFO [M:0;10.0.0.110:51155] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.0.0.110%2C51155%2C1434564538919.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51155,1434564538919, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:08:59,003 INFO [M:0;10.0.0.110:51155] wal.FSHLog(1365): Slow sync cost: 3 ms, current pipeline: [] 2015-06-17 11:08:59,003 INFO [M:0;10.0.0.110:51155] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51155,1434564538919/10.0.0.110%2C51155%2C1434564538919.default.1434564538998 2015-06-17 11:08:59,003 INFO [M:0;10.0.0.110:51155] regionserver.MetricsRegionServerWrapperImpl(113): Computing regionserver metrics every 5000 milliseconds 2015-06-17 11:08:59,004 DEBUG [M:0;10.0.0.110:51155] executor.ExecutorService(91): Starting executor service name=RS_OPEN_REGION-10.0.0.110:51155, corePoolSize=3, maxPoolSize=3 2015-06-17 11:08:59,004 DEBUG [M:0;10.0.0.110:51155] executor.ExecutorService(91): Starting executor service name=RS_OPEN_META-10.0.0.110:51155, corePoolSize=1, maxPoolSize=1 2015-06-17 11:08:59,004 DEBUG [M:0;10.0.0.110:51155] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_REGION-10.0.0.110:51155, corePoolSize=3, maxPoolSize=3 2015-06-17 11:08:59,004 DEBUG [M:0;10.0.0.110:51155] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_META-10.0.0.110:51155, corePoolSize=1, maxPoolSize=1 2015-06-17 11:08:59,004 DEBUG [M:0;10.0.0.110:51155] executor.ExecutorService(91): Starting executor service name=RS_LOG_REPLAY_OPS-10.0.0.110:51155, corePoolSize=2, maxPoolSize=2 2015-06-17 11:08:59,004 DEBUG [M:0;10.0.0.110:51155] executor.ExecutorService(91): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-10.0.0.110:51155, corePoolSize=3, maxPoolSize=3 2015-06-17 11:08:59,005 DEBUG [M:0;10.0.0.110:51155] zookeeper.ZKUtil(486): master:51155-0x14e02b3a2320031, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:08:59,006 DEBUG [M:0;10.0.0.110:51155] zookeeper.ZKUtil(486): master:51155-0x14e02b3a2320031, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51155,1434564538919 2015-06-17 11:08:59,006 INFO [M:0;10.0.0.110:51155] regionserver.ReplicationSourceManager(229): Current list of replicators: [10.0.0.110,50905,1434564404764, 10.0.0.110,51155,1434564538919] other RSs: [10.0.0.110,50905,1434564404764, 10.0.0.110,51155,1434564538919] 2015-06-17 11:08:59,018 INFO [M:0;10.0.0.110:51155] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x4f5772b7 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:08:59,020 DEBUG [M:0;10.0.0.110:51155-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x4f5772b70x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:08:59,021 DEBUG [M:0;10.0.0.110:51155] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodecWithTags@9304030, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:08:59,021 DEBUG [M:0;10.0.0.110:51155] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:08:59,021 DEBUG [M:0;10.0.0.110:51155] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:08:59,022 DEBUG [M:0;10.0.0.110:51155-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x4f5772b7-0x14e02b3a2320035 connected 2015-06-17 11:08:59,026 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=170, retries=350, started=46011 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:59,029 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=311, retries=350, started=75461 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:59,033 INFO [M:0;10.0.0.110:51155] regionserver.HeapMemoryManager(190): Starting HeapMemoryTuner chore. 2015-06-17 11:08:59,033 INFO [SplitLogWorker-10.0.0.110:51155] regionserver.SplitLogWorker(134): SplitLogWorker 10.0.0.110,51155,1434564538919 starting 2015-06-17 11:08:59,033 INFO [M:0;10.0.0.110:51155] regionserver.HRegionServer(1363): Serving as 10.0.0.110,51155,1434564538919, RpcServer on 10.0.0.110/10.0.0.110:51155, sessionid=0x14e02b3a2320031 2015-06-17 11:08:59,033 DEBUG [M:0;10.0.0.110:51155] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2015-06-17 11:08:59,033 DEBUG [M:0;10.0.0.110:51155] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager 10.0.0.110,51155,1434564538919 2015-06-17 11:08:59,033 DEBUG [M:0;10.0.0.110:51155] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,51155,1434564538919' 2015-06-17 11:08:59,033 DEBUG [M:0;10.0.0.110:51155] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2015-06-17 11:08:59,034 DEBUG [M:0;10.0.0.110:51155] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2015-06-17 11:08:59,034 DEBUG [M:0;10.0.0.110:51155] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2015-06-17 11:08:59,034 DEBUG [M:0;10.0.0.110:51155] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2015-06-17 11:08:59,034 DEBUG [M:0;10.0.0.110:51155] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager 10.0.0.110,51155,1434564538919 2015-06-17 11:08:59,034 DEBUG [M:0;10.0.0.110:51155] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,51155,1434564538919' 2015-06-17 11:08:59,034 DEBUG [M:0;10.0.0.110:51155] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2015-06-17 11:08:59,034 DEBUG [M:0;10.0.0.110:51155] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2015-06-17 11:08:59,035 DEBUG [M:0;10.0.0.110:51155] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2015-06-17 11:08:59,035 INFO [M:0;10.0.0.110:51155] quotas.RegionServerQuotaManager(62): Quota support disabled 2015-06-17 11:08:59,043 INFO [10.0.0.110:51155.activeMasterManager] master.ServerManager(1028): Finished waiting for region servers count to settle; checked in 1, slept for 51 ms, expecting minimum of 1, maximum of 1, master is running 2015-06-17 11:08:59,043 INFO [10.0.0.110:51155.activeMasterManager] master.ServerManager(459): Registering server=10.0.0.110,50905,1434564404764 2015-06-17 11:08:59,043 INFO [10.0.0.110:51155.activeMasterManager] master.HMaster(665): Registered server found up in zk but who has not yet reported in: 10.0.0.110,50905,1434564404764 2015-06-17 11:08:59,045 INFO [10.0.0.110:51155.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50905,1434564404764 belongs to an existing region server 2015-06-17 11:08:59,048 INFO [10.0.0.110:51155.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51155,1434564538919 belongs to an existing region server 2015-06-17 11:08:59,049 WARN [10.0.0.110:51155.activeMasterManager] master.RegionStates(339): Open region's last host null should be the same as the current one 10.0.0.110,51130,1434564521571, ignored the last and used the current one 2015-06-17 11:08:59,096 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:08:59,098 DEBUG [AsyncRpcChannel-pool2-t3] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service MasterService, sasl=false 2015-06-17 11:08:59,098 DEBUG [RpcServer.listener,port=51155] ipc.RpcServer$Listener(840): RpcServer.listener,port=51155: connection from 10.0.0.110:51164; # active connections: 1 2015-06-17 11:08:59,099 INFO [RpcServer.reader=1,bindAddress=10.0.0.110,port=51155] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51164 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:08:59,100 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:59,100 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51155: callId: 1781 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:59,100 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1725 ms ago, cancelled=false, msg= 2015-06-17 11:08:59,143 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=350, started=17188 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:08:59,228 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=171, retries=350, started=46213 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:59,230 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=312, retries=350, started=75662 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:59,301 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:59,301 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51155: callId: 1783 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:59,302 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1927 ms ago, cancelled=false, msg= 2015-06-17 11:08:59,344 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=350, started=17389 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:08:59,430 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=172, retries=350, started=46415 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:59,430 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=313, retries=350, started=75862 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:59,505 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:59,505 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51155: callId: 1785 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:59,505 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2130 ms ago, cancelled=false, msg= 2015-06-17 11:08:59,544 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=350, started=17589 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:08:59,630 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=173, retries=350, started=46615 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:59,631 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=314, retries=350, started=76063 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:59,707 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:59,707 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51155: callId: 1787 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:59,707 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2332 ms ago, cancelled=false, msg= 2015-06-17 11:08:59,746 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=350, started=17791 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:08:59,831 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=174, retries=350, started=46816 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:08:59,832 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=315, retries=350, started=76264 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:08:59,910 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:08:59,910 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51155: callId: 1789 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:08:59,910 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2535 ms ago, cancelled=false, msg= 2015-06-17 11:08:59,948 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=350, started=17993 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:00,031 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=175, retries=350, started=47016 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:00,033 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=316, retries=350, started=76465 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:00,112 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:00,112 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51155: callId: 1791 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:00,112 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2737 ms ago, cancelled=false, msg= 2015-06-17 11:09:00,149 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=350, started=18194 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:00,232 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=176, retries=350, started=47217 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:00,233 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=317, retries=350, started=76665 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:00,315 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:00,315 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51155: callId: 1793 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:00,315 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2940 ms ago, cancelled=false, msg= 2015-06-17 11:09:00,349 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=350, started=18394 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:00,433 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=177, retries=350, started=47418 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:00,434 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=318, retries=350, started=76866 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:00,517 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:00,517 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51155: callId: 1795 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:00,518 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3143 ms ago, cancelled=false, msg= 2015-06-17 11:09:00,549 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=350, started=18594 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:00,558 DEBUG [AsyncRpcChannel-pool2-t4] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2015-06-17 11:09:00,559 DEBUG [RpcServer.listener,port=51155] ipc.RpcServer$Listener(840): RpcServer.listener,port=51155: connection from 10.0.0.110:51165; # active connections: 2 2015-06-17 11:09:00,560 INFO [RpcServer.reader=2,bindAddress=10.0.0.110,port=51155] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51165 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:09:00,636 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=178, retries=350, started=47621 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:00,636 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=319, retries=350, started=77068 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:00,720 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:00,720 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51155: callId: 1797 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:00,720 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3345 ms ago, cancelled=false, msg= 2015-06-17 11:09:00,750 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=350, started=18795 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:00,837 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=179, retries=350, started=47822 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:00,837 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=320, retries=350, started=77269 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:00,922 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:00,922 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51155: callId: 1799 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:00,922 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3547 ms ago, cancelled=false, msg= 2015-06-17 11:09:00,952 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=350, started=18997 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:01,037 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=180, retries=350, started=48022 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:01,038 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=321, retries=350, started=77470 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:01,124 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:01,124 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51155: callId: 1801 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:01,125 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3750 ms ago, cancelled=false, msg= 2015-06-17 11:09:01,153 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=350, started=19198 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:01,238 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=181, retries=350, started=48223 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:01,238 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=322, retries=350, started=77670 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:01,327 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:01,327 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51155: callId: 1803 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:01,328 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3953 ms ago, cancelled=false, msg= 2015-06-17 11:09:01,353 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=350, started=19398 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:01,439 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=182, retries=350, started=48424 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:01,440 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=323, retries=350, started=77872 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:01,530 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:01,530 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51155: callId: 1805 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:01,530 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4155 ms ago, cancelled=false, msg= 2015-06-17 11:09:01,554 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=350, started=19599 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:01,640 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=183, retries=350, started=48625 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:01,640 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=324, retries=350, started=78072 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:01,731 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:01,731 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51155: callId: 1807 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:01,732 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4357 ms ago, cancelled=false, msg= 2015-06-17 11:09:01,755 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=350, started=19800 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:01,842 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=184, retries=350, started=48827 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:01,842 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=325, retries=350, started=78274 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:01,933 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:01,933 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51155: callId: 1809 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:01,933 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4558 ms ago, cancelled=false, msg= 2015-06-17 11:09:01,956 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=350, started=20001 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:02,043 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=185, retries=350, started=49028 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:02,043 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=326, retries=350, started=78475 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:02,135 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:02,135 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51155: callId: 1811 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:02,135 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4760 ms ago, cancelled=false, msg= 2015-06-17 11:09:02,158 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=350, started=20203 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:02,244 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=186, retries=350, started=49229 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:02,245 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=327, retries=350, started=78677 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:02,338 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:02,339 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51155: callId: 1813 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:02,339 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4964 ms ago, cancelled=false, msg= 2015-06-17 11:09:02,359 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=350, started=20404 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:02,445 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=328, retries=350, started=78877 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:02,445 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=187, retries=350, started=49430 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:02,540 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:02,540 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51155: callId: 1815 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:02,541 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5166 ms ago, cancelled=false, msg= 2015-06-17 11:09:02,561 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=350, started=20606 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:02,646 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=329, retries=350, started=79078 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:02,646 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=188, retries=350, started=49631 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:02,743 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:02,743 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51155: callId: 1817 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:02,743 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5368 ms ago, cancelled=false, msg= 2015-06-17 11:09:02,743 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: Can't get master address from ZooKeeper; znode data == null Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564537375, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:09:02,762 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=350, started=20807 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:02,846 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=189, retries=350, started=49831 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:02,848 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=330, retries=350, started=79280 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:02,945 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:02,945 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51155: callId: 1819 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:02,948 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:02,948 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51155: callId: 1821 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:02,952 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:02,952 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51155: callId: 1823 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:02,959 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:02,959 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51155: callId: 1825 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:02,963 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=350, started=21008 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:02,970 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:02,970 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51155: callId: 1827 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:02,992 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:02,992 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51155: callId: 1829 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:03,034 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:03,034 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51155: callId: 1831 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:03,048 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=190, retries=350, started=50033 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:03,050 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=331, retries=350, started=79482 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:03,135 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:03,135 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51155: callId: 1833 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:03,163 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=350, started=21208 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:03,237 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:03,237 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51155: callId: 1835 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:03,248 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=191, retries=350, started=50233 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:03,250 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=332, retries=350, started=79682 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:03,339 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:03,339 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51155: callId: 1837 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:03,364 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=350, started=21409 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:03,440 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:03,440 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51155: callId: 1839 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:03,441 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=497 ms ago, cancelled=false, msg= 2015-06-17 11:09:03,450 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=192, retries=350, started=50435 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:03,452 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=333, retries=350, started=79884 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:03,566 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=350, started=21611 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:03,643 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:03,643 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51155: callId: 1841 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:03,643 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=699 ms ago, cancelled=false, msg= 2015-06-17 11:09:03,651 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=193, retries=350, started=50636 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:03,653 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=334, retries=350, started=80085 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:03,767 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=350, started=21812 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:03,845 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:03,845 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51155: callId: 1843 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:03,846 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=902 ms ago, cancelled=false, msg= 2015-06-17 11:09:03,852 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=194, retries=350, started=50837 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:03,854 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=335, retries=350, started=80286 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:03,969 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=35, retries=350, started=22014 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:04,047 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:04,047 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51155: callId: 1845 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:04,047 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1103 ms ago, cancelled=false, msg= 2015-06-17 11:09:04,053 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=195, retries=350, started=51038 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:04,054 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=336, retries=350, started=80486 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:04,169 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=36, retries=350, started=22214 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:04,248 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:04,248 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51155: callId: 1847 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:04,249 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1305 ms ago, cancelled=false, msg= 2015-06-17 11:09:04,255 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=337, retries=350, started=80687 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:04,255 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=196, retries=350, started=51240 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:04,370 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=37, retries=350, started=22415 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:04,451 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:04,451 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51155: callId: 1849 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:04,452 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1508 ms ago, cancelled=false, msg= 2015-06-17 11:09:04,455 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=338, retries=350, started=80887 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:04,456 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=197, retries=350, started=51441 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:04,572 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=38, retries=350, started=22617 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:04,654 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:04,654 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51155: callId: 1851 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:04,654 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1710 ms ago, cancelled=false, msg= 2015-06-17 11:09:04,657 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=339, retries=350, started=81089 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:04,657 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=198, retries=350, started=51642 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:04,773 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=39, retries=350, started=22818 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:04,856 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:04,856 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51155: callId: 1853 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:04,857 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=340, retries=350, started=81289 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:04,857 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1913 ms ago, cancelled=false, msg= 2015-06-17 11:09:04,858 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=199, retries=350, started=51843 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:04,974 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=40, retries=350, started=23019 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:05,058 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=341, retries=350, started=81490 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:05,058 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=200, retries=350, started=52043 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:05,059 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:05,059 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51155: callId: 1855 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:05,059 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2115 ms ago, cancelled=false, msg= 2015-06-17 11:09:05,174 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=41, retries=350, started=23219 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:05,258 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=201, retries=350, started=52243 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:05,260 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=342, retries=350, started=81692 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:05,262 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:05,263 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51155: callId: 1857 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:05,263 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2319 ms ago, cancelled=false, msg= 2015-06-17 11:09:05,375 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=42, retries=350, started=23420 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:05,459 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=202, retries=350, started=52444 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:05,461 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=343, retries=350, started=81893 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:05,466 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:05,466 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51155: callId: 1859 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:05,466 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2522 ms ago, cancelled=false, msg= 2015-06-17 11:09:05,577 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=43, retries=350, started=23622 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:05,660 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=203, retries=350, started=52645 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:05,663 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=344, retries=350, started=82095 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:05,670 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:05,670 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51155: callId: 1861 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:05,670 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2726 ms ago, cancelled=false, msg= 2015-06-17 11:09:05,780 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=44, retries=350, started=23825 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:05,860 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=204, retries=350, started=52845 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:05,864 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=345, retries=350, started=82296 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:05,871 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:05,871 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51155: callId: 1863 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:05,872 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2928 ms ago, cancelled=false, msg= 2015-06-17 11:09:05,981 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=45, retries=350, started=24026 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:06,062 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=205, retries=350, started=53047 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:06,064 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=346, retries=350, started=82496 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:06,073 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:06,073 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51155: callId: 1865 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:06,073 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3129 ms ago, cancelled=false, msg= 2015-06-17 11:09:06,181 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=46, retries=350, started=24226 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:06,264 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=206, retries=350, started=53249 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:06,266 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=347, retries=350, started=82698 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:06,275 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:06,275 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51155: callId: 1867 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:06,275 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3331 ms ago, cancelled=false, msg= 2015-06-17 11:09:06,381 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=47, retries=350, started=24426 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:06,464 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=207, retries=350, started=53449 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:06,466 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=348, retries=350, started=82898 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:06,478 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:06,478 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51155: callId: 1869 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:06,478 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3534 ms ago, cancelled=false, msg= 2015-06-17 11:09:06,582 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=48, retries=350, started=24627 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:06,664 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=208, retries=350, started=53649 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:06,667 INFO [10.0.0.110:51015.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=349, retries=350, started=83099 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,50975,1434564429984, seqNum=12 2015-06-17 11:09:06,670 WARN [10.0.0.110:51015.activeMasterManager] master.TableNamespaceManager(292): Caught exception in initializing namespace table manager org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=350, exceptions: Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call id=32 on server /10.0.0.110:50975 aborted: connection is closing Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:721) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:687) at org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:137) at org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:268) at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:98) at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:919) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:749) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1542) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: hconnection-0x62523e8d closed at org.apache.hadoop.hbase.client.ConnectionImplementation.getTableState(ConnectionImplementation.java:2235) at org.apache.hadoop.hbase.client.ConnectionImplementation.isTableDisabled(ConnectionImplementation.java:553) at org.apache.hadoop.hbase.client.ConnectionImplementation.relocateRegion(ConnectionImplementation.java:718) at org.apache.hadoop.hbase.client.ConnectionImplementation.relocateRegion(ConnectionImplementation.java:706) at org.apache.hadoop.hbase.client.ConnectionImplementation.getRegionLocation(ConnectionImplementation.java:531) at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:81) at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:79) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) ... 10 more 2015-06-17 11:09:06,671 FATAL [10.0.0.110:51015.activeMasterManager] master.HMaster$1(1546): Failed to become active master org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=350, exceptions: Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call id=32 on server /10.0.0.110:50975 aborted: connection is closing Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:07:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564463568, pause=1, retries=350}, java.io.IOException: hconnection-0x62523e8d closed at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:721) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:687) at org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:137) at org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:268) at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:98) at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:919) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:749) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1542) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: hconnection-0x62523e8d closed at org.apache.hadoop.hbase.client.ConnectionImplementation.getTableState(ConnectionImplementation.java:2235) at org.apache.hadoop.hbase.client.ConnectionImplementation.isTableDisabled(ConnectionImplementation.java:553) at org.apache.hadoop.hbase.client.ConnectionImplementation.relocateRegion(ConnectionImplementation.java:718) at org.apache.hadoop.hbase.client.ConnectionImplementation.relocateRegion(ConnectionImplementation.java:706) at org.apache.hadoop.hbase.client.ConnectionImplementation.getRegionLocation(ConnectionImplementation.java:531) at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:81) at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:79) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) ... 10 more 2015-06-17 11:09:06,680 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:06,680 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51155: callId: 1871 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:06,680 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3736 ms ago, cancelled=false, msg= 2015-06-17 11:09:06,783 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=49, retries=350, started=24828 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:06,866 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=209, retries=350, started=53851 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:06,881 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:06,881 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51155: callId: 1873 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:06,881 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3937 ms ago, cancelled=false, msg= 2015-06-17 11:09:06,984 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=50, retries=350, started=25029 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:07,067 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=210, retries=350, started=54052 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:07,082 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:07,083 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51155: callId: 1875 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:07,083 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4139 ms ago, cancelled=false, msg= 2015-06-17 11:09:07,184 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=51, retries=350, started=25229 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:07,267 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=211, retries=350, started=54252 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:07,284 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:07,284 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51155: callId: 1877 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:07,284 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4340 ms ago, cancelled=false, msg= 2015-06-17 11:09:07,385 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=52, retries=350, started=25430 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:07,468 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=212, retries=350, started=54453 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:07,487 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:07,487 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51155: callId: 1879 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:07,487 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4543 ms ago, cancelled=false, msg= 2015-06-17 11:09:07,587 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=53, retries=350, started=25632 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:07,670 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=213, retries=350, started=54655 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:07,690 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:07,690 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51155: callId: 1881 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:07,691 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4747 ms ago, cancelled=false, msg= 2015-06-17 11:09:07,787 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=54, retries=350, started=25832 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:07,872 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=214, retries=350, started=54857 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:07,893 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:07,893 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51155: callId: 1883 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:07,893 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4949 ms ago, cancelled=false, msg= 2015-06-17 11:09:07,987 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=55, retries=350, started=26032 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:08,073 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=215, retries=350, started=55058 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:08,095 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:08,095 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51155: callId: 1885 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:08,096 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5152 ms ago, cancelled=false, msg= 2015-06-17 11:09:08,189 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=56, retries=350, started=26234 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:08,274 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=216, retries=350, started=55259 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:08,296 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:08,297 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51155: callId: 1887 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:08,297 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5353 ms ago, cancelled=false, msg= 2015-06-17 11:09:08,297 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564542944, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:09:08,390 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=57, retries=350, started=26435 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:08,474 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=217, retries=350, started=55459 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:08,499 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:08,499 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51155: callId: 1889 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:08,502 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:08,502 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51155: callId: 1891 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:08,507 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:08,507 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51155: callId: 1893 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:08,513 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:08,514 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51155: callId: 1895 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:08,525 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:08,525 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51155: callId: 1897 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:08,546 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:08,546 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51155: callId: 1899 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:08,588 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:08,588 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51155: callId: 1901 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:08,592 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=58, retries=350, started=26637 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:08,677 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=218, retries=350, started=55662 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:08,690 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:08,690 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51155: callId: 1903 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:08,791 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:08,791 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51155: callId: 1905 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:08,792 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=59, retries=350, started=26837 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:08,877 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=219, retries=350, started=55862 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:08,893 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:08,893 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51155: callId: 1907 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:08,972 DEBUG [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(188): Sleeping for 3500 before executing next master restart count=9 2015-06-17 11:09:08,993 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=60, retries=350, started=27038 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:08,994 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:08,994 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51155: callId: 1909 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:08,994 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=496 ms ago, cancelled=false, msg= 2015-06-17 11:09:09,078 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=220, retries=350, started=56063 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:09,195 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=61, retries=350, started=27240 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:09,196 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:09,196 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51155: callId: 1911 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:09,197 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=699 ms ago, cancelled=false, msg= 2015-06-17 11:09:09,278 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=221, retries=350, started=56263 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:09,396 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=62, retries=350, started=27440 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:09,398 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:09,398 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51155: callId: 1913 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:09,398 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=900 ms ago, cancelled=false, msg= 2015-06-17 11:09:09,480 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=222, retries=350, started=56465 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:09,596 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=63, retries=350, started=27641 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:09,600 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:09,601 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51155: callId: 1915 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:09,601 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1103 ms ago, cancelled=false, msg= 2015-06-17 11:09:09,680 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=223, retries=350, started=56665 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:09,796 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=64, retries=350, started=27841 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:09,803 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:09,803 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51155: callId: 1917 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:09,803 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1305 ms ago, cancelled=false, msg= 2015-06-17 11:09:09,881 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=224, retries=350, started=56866 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:09,997 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=65, retries=350, started=28042 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:10,006 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:10,006 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51155: callId: 1919 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:10,007 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1509 ms ago, cancelled=false, msg= 2015-06-17 11:09:10,082 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=225, retries=350, started=57067 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:10,199 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=66, retries=350, started=28244 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:10,209 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:10,209 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51155: callId: 1921 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:10,209 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1711 ms ago, cancelled=false, msg= 2015-06-17 11:09:10,284 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=226, retries=350, started=57269 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:10,399 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=67, retries=350, started=28444 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:10,410 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:10,410 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51155: callId: 1923 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:10,410 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1912 ms ago, cancelled=false, msg= 2015-06-17 11:09:10,485 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=227, retries=350, started=57470 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:10,599 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=68, retries=350, started=28644 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:10,611 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:10,612 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51155: callId: 1925 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:10,612 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2114 ms ago, cancelled=false, msg= 2015-06-17 11:09:10,685 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=228, retries=350, started=57670 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:10,800 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=69, retries=350, started=28845 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:10,814 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:10,814 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51155: callId: 1927 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:10,814 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2316 ms ago, cancelled=false, msg= 2015-06-17 11:09:10,887 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=229, retries=350, started=57872 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:11,001 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=70, retries=350, started=29046 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:11,017 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:11,017 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51155: callId: 1929 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:11,018 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2520 ms ago, cancelled=false, msg= 2015-06-17 11:09:11,087 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=230, retries=350, started=58072 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:11,203 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=71, retries=350, started=29248 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:11,221 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:11,221 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51155: callId: 1931 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:11,221 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2723 ms ago, cancelled=false, msg= 2015-06-17 11:09:11,287 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=231, retries=350, started=58272 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:11,403 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=72, retries=350, started=29448 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:11,422 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:11,422 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51155: callId: 1933 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:11,423 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2925 ms ago, cancelled=false, msg= 2015-06-17 11:09:11,490 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=232, retries=350, started=58475 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:11,604 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=73, retries=350, started=29649 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:11,624 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:11,624 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51155: callId: 1935 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:11,624 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3126 ms ago, cancelled=false, msg= 2015-06-17 11:09:11,690 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=233, retries=350, started=58675 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:11,806 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=74, retries=350, started=29851 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:11,827 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:11,827 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51155: callId: 1937 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:11,827 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3329 ms ago, cancelled=false, msg= 2015-06-17 11:09:11,891 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=234, retries=350, started=58876 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:12,007 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=75, retries=350, started=30052 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:12,029 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:12,029 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51155: callId: 1939 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:12,029 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3531 ms ago, cancelled=false, msg= 2015-06-17 11:09:12,093 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=235, retries=350, started=59078 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:12,208 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=76, retries=350, started=30253 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:12,230 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:12,231 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51155: callId: 1941 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:12,231 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3733 ms ago, cancelled=false, msg= 2015-06-17 11:09:12,294 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=236, retries=350, started=59279 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:12,409 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=77, retries=350, started=30454 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:12,433 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:12,433 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51155: callId: 1943 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:12,433 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3935 ms ago, cancelled=false, msg= 2015-06-17 11:09:12,473 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(197): executing master kill 2015-06-17 11:09:12,473 INFO [MasterRestartThread] hbase.MiniHBaseCluster(422): Aborting 10.0.0.110,51155,1434564538919 2015-06-17 11:09:12,473 FATAL [MasterRestartThread] master.HMaster(1953): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2015-06-17 11:09:12,473 FATAL [MasterRestartThread] master.HMaster(1956): Aborting for tests java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:423) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:270) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$5.run(TestMasterRestartWithProcedures.java:198) 2015-06-17 11:09:12,473 INFO [MasterRestartThread] regionserver.HRegionServer(1859): STOPPED: Aborting for tests 2015-06-17 11:09:12,473 INFO [MasterRestartThread] hbase.LocalHBaseCluster(379): Waiting on 10.0.0.110,51155,1434564538919 2015-06-17 11:09:12,473 INFO [M:0;10.0.0.110:51155] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2015-06-17 11:09:12,473 INFO [M:0;10.0.0.110:51155] regionserver.HeapMemoryManager(201): Stoping HeapMemoryTuner chore. 2015-06-17 11:09:12,473 INFO [SplitLogWorker-10.0.0.110:51155] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2015-06-17 11:09:12,473 INFO [SplitLogWorker-10.0.0.110:51155] regionserver.SplitLogWorker(155): SplitLogWorker 10.0.0.110,51155,1434564538919 exiting 2015-06-17 11:09:12,474 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2015-06-17 11:09:12,473 INFO [M:0;10.0.0.110:51155] procedure2.ProcedureExecutor(490): Stopping the procedure executor 2015-06-17 11:09:12,474 INFO [M:0;10.0.0.110:51155] wal.WALProcedureStore(159): Stopping the WAL Procedure Store 2015-06-17 11:09:12,473 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2015-06-17 11:09:12,480 INFO [IPC Server handler 4 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741888_1064{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 0 2015-06-17 11:09:12,481 INFO [M:0;10.0.0.110:51155] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2015-06-17 11:09:12,481 INFO [M:0;10.0.0.110:51155] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2015-06-17 11:09:12,481 INFO [M:0;10.0.0.110:51155] regionserver.HRegionServer(1021): stopping server 10.0.0.110,51155,1434564538919 2015-06-17 11:09:12,481 DEBUG [M:0;10.0.0.110:51155] zookeeper.MetaTableLocator(617): Stopping MetaTableLocator 2015-06-17 11:09:12,481 INFO [M:0;10.0.0.110:51155] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320032 2015-06-17 11:09:12,482 DEBUG [M:0;10.0.0.110:51155] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:09:12,482 INFO [M:0;10.0.0.110:51155] regionserver.HRegionServer(1049): stopping server 10.0.0.110,51155,1434564538919; all regions closed. 2015-06-17 11:09:12,482 DEBUG [M:0;10.0.0.110:51155] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51155,1434564538919 2015-06-17 11:09:12,482 INFO [10.0.0.110:51155.activeMasterManager] zookeeper.MetaTableLocator(340): Failed verification of hbase:meta,,1 at address=10.0.0.110,51130,1434564521571, exception=Call id=0 on server /10.0.0.110:51130 aborted: connection is closing 2015-06-17 11:09:12,482 DEBUG [AsyncRpcChannel-pool2-t2] ipc.AsyncRpcChannel$8(574): IPC Client (-580076493) connection to /10.0.0.110:51130 from ndimiduk.hfs.9: closed 2015-06-17 11:09:12,483 DEBUG [10.0.0.110:51155.activeMasterManager] zookeeper.ZKUtil(745): master:51155-0x14e02b3a2320031, quorum=localhost:51848, baseZNode=/hbase Unable to get data of znode /hbase/recovering-regions/1588230740 because node does not exist (not an error) 2015-06-17 11:09:12,484 INFO [IPC Server handler 3 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741887_1063{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 83 2015-06-17 11:09:12,484 DEBUG [10.0.0.110:51155.activeMasterManager] coordination.ZKSplitLogManagerCoordination(650): Marked 1588230740 recovering from 10.0.0.110,51130,1434564521571: /hbase/recovering-regions/1588230740/10.0.0.110,51130,1434564521571 2015-06-17 11:09:12,484 INFO [10.0.0.110:51155.activeMasterManager] master.HMaster(884): Re-assigning hbase:meta with replicaId, 0 it was on 10.0.0.110,51130,1434564521571 2015-06-17 11:09:12,485 INFO [10.0.0.110:51155.activeMasterManager] master.RegionStates(1072): Transition {1588230740 state=OPEN, ts=1434564539049, server=10.0.0.110,51130,1434564521571} to {1588230740 state=OFFLINE, ts=1434564552485, server=10.0.0.110,51130,1434564521571} 2015-06-17 11:09:12,485 INFO [10.0.0.110:51155.activeMasterManager] zookeeper.MetaTableLocator(440): Setting hbase:meta region location in ZooKeeper as 10.0.0.110,51130,1434564521571 2015-06-17 11:09:12,485 INFO [10.0.0.110:51155.activeMasterManager] master.AssignmentManager(996): Skip assigning hbase:meta,,1.1588230740, the server is stopped/aborted 2015-06-17 11:09:12,485 FATAL [10.0.0.110:51155.activeMasterManager] master.HMaster$1(1546): Failed to become active master java.io.IOException: connection is closed at org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:254) at org.apache.hadoop.hbase.MetaTableAccessor.putToMetaTable(MetaTableAccessor.java:1341) at org.apache.hadoop.hbase.MetaTableAccessor.updateTableState(MetaTableAccessor.java:1691) at org.apache.hadoop.hbase.MetaTableAccessor.updateTableState(MetaTableAccessor.java:1110) at org.apache.hadoop.hbase.master.TableStateManager.udpateMetaState(TableStateManager.java:180) at org.apache.hadoop.hbase.master.TableStateManager.setTableState(TableStateManager.java:67) at org.apache.hadoop.hbase.master.HMaster.assignMeta(HMaster.java:892) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:711) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1542) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:12,487 DEBUG [M:0;10.0.0.110:51155] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:09:12,487 INFO [M:0;10.0.0.110:51155] wal.FSHLog(977): Closed WAL: FSHLog 10.0.0.110%2C51155%2C1434564538919.default:(num 1434564538998) 2015-06-17 11:09:12,487 DEBUG [M:0;10.0.0.110:51155] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:09:12,487 INFO [M:0;10.0.0.110:51155] regionserver.Leases(146): M:0;10.0.0.110:51155 closing leases 2015-06-17 11:09:12,487 INFO [M:0;10.0.0.110:51155] regionserver.Leases(149): M:0;10.0.0.110:51155 closed leases 2015-06-17 11:09:12,487 INFO [M:0;10.0.0.110:51155] hbase.ChoreService(302): Chore service for: 10.0.0.110,51155,1434564538919 had [[ScheduledChore: Name: 10.0.0.110,51155,1434564538919-MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region 10.0.0.110,51155,1434564538919 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS]] on shutdown 2015-06-17 11:09:12,497 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=237, retries=350, started=59482 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:12,611 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=78, retries=350, started=30656 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:12,637 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:12,638 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51155: callId: 1945 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:12,638 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4140 ms ago, cancelled=false, msg= 2015-06-17 11:09:12,699 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=238, retries=350, started=59684 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:12,814 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=79, retries=350, started=30859 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:12,842 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:12,842 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51155] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51155: callId: 1947 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51164 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:12,843 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4345 ms ago, cancelled=false, msg= 2015-06-17 11:09:12,900 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=239, retries=350, started=59884 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:12,954 INFO [10.0.0.110,51155,1434564538919_splitLogManager__ChoreService_1] hbase.ScheduledChore(179): Chore: SplitLogManager Timeout Monitor was stopped 2015-06-17 11:09:13,010 INFO [master//10.0.0.110:0.logRoller] regionserver.LogRoller(159): LogRoller exiting. 2015-06-17 11:09:13,010 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(146): master//10.0.0.110:0.leaseChecker closing leases 2015-06-17 11:09:13,010 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(149): master//10.0.0.110:0.leaseChecker closed leases 2015-06-17 11:09:13,010 INFO [M:0;10.0.0.110:51155] regionserver.CompactSplitThread(394): Waiting for Split Thread to finish... 2015-06-17 11:09:13,011 INFO [M:0;10.0.0.110:51155] regionserver.CompactSplitThread(394): Waiting for Merge Thread to finish... 2015-06-17 11:09:13,011 INFO [M:0;10.0.0.110:51155] regionserver.CompactSplitThread(394): Waiting for Large Compaction Thread to finish... 2015-06-17 11:09:13,011 INFO [M:0;10.0.0.110:51155] regionserver.CompactSplitThread(394): Waiting for Small Compaction Thread to finish... 2015-06-17 11:09:13,014 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=80, retries=350, started=31059 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:13,014 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51155-0x14e02b3a2320031, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/10.0.0.110,51155,1434564538919 2015-06-17 11:09:13,014 INFO [M:0;10.0.0.110:51155] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320035 2015-06-17 11:09:13,015 DEBUG [M:0;10.0.0.110:51155] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:09:13,016 DEBUG [M:0;10.0.0.110:51155] master.HMaster(1074): Stopping service threads 2015-06-17 11:09:13,017 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:09:13,017 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51155-0x14e02b3a2320031, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:09:13,017 INFO [M:0;10.0.0.110:51155] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320033 2015-06-17 11:09:13,017 DEBUG [main-EventThread] zookeeper.ZKUtil(488): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:09:13,017 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(488): master:51155-0x14e02b3a2320031, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:09:13,018 DEBUG [M:0;10.0.0.110:51155] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:09:13,018 INFO [M:0;10.0.0.110:51155] hbase.ChoreService(302): Chore service for: 10.0.0.110,51155,1434564538919_splitLogManager_ had [] on shutdown 2015-06-17 11:09:13,018 INFO [M:0;10.0.0.110:51155] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2015-06-17 11:09:13,018 INFO [M:0;10.0.0.110:51155] ipc.RpcServer(2219): Stopping server on 51155 2015-06-17 11:09:13,018 INFO [RpcServer.listener,port=51155] ipc.RpcServer$Listener(776): RpcServer.listener,port=51155: stopping 2015-06-17 11:09:13,018 INFO [RpcServer.responder] ipc.RpcServer$Responder(1018): RpcServer.responder: stopped 2015-06-17 11:09:13,019 INFO [RpcServer.responder] ipc.RpcServer$Responder(921): RpcServer.responder: stopping 2015-06-17 11:09:13,020 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,51155,1434564538919 2015-06-17 11:09:13,020 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51155-0x14e02b3a2320031, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,51155,1434564538919 2015-06-17 11:09:13,020 INFO [MasterRestartThread-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [10.0.0.110,51155,1434564538919] 2015-06-17 11:09:13,020 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51155-0x14e02b3a2320031, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:09:13,020 INFO [M:0;10.0.0.110:51155] regionserver.HRegionServer(1093): stopping server 10.0.0.110,51155,1434564538919; zookeeper connection closed. 2015-06-17 11:09:13,020 INFO [M:0;10.0.0.110:51155] regionserver.HRegionServer(1096): M:0;10.0.0.110:51155 exiting 2015-06-17 11:09:13,020 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(200): executing master restart 2015-06-17 11:09:13,021 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:13,021 INFO [main-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(121): /hbase/rs/10.0.0.110,51155,1434564538919 znode expired, triggering replicatorRemoved event 2015-06-17 11:09:13,021 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:09:13,021 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:13,033 INFO [MasterRestartThread] client.ConnectionUtils(105): master//10.0.0.110:0 server-side HConnection retries=350 2015-06-17 11:09:13,034 INFO [MasterRestartThread] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2015-06-17 11:09:13,036 INFO [MasterRestartThread] ipc.RpcServer$Listener(598): master//10.0.0.110:0: started 10 reader(s). 2015-06-17 11:09:13,037 INFO [MasterRestartThread] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:09:13,038 INFO [MasterRestartThread] zookeeper.RecoverableZooKeeper(121): Process identifier=master:51168 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:09:13,040 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:511680x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:09:13,041 DEBUG [MasterRestartThread] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-17 11:09:13,041 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(578): master:51168-0x14e02b3a2320036 connected 2015-06-17 11:09:13,043 DEBUG [MasterRestartThread] zookeeper.ZKUtil(488): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:09:13,043 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2015-06-17 11:09:13,043 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:09:13,044 INFO [RpcServer.listener,port=51168] ipc.RpcServer$Listener(729): RpcServer.listener,port=51168: starting 2015-06-17 11:09:13,044 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=0 queue=0 2015-06-17 11:09:13,043 INFO [RpcServer.responder] ipc.RpcServer$Responder(917): RpcServer.responder: starting 2015-06-17 11:09:13,044 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=1 queue=0 2015-06-17 11:09:13,044 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=2 queue=0 2015-06-17 11:09:13,044 DEBUG [Thread-199] zookeeper.ZKUtil(745): hconnection-0x51e5f22a-0x14e02b3a2320008, quorum=localhost:51848, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2015-06-17 11:09:13,044 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=3 queue=0 2015-06-17 11:09:13,044 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4546 ms ago, cancelled=false, msg= 2015-06-17 11:09:13,044 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=4 queue=0 2015-06-17 11:09:13,044 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=0 queue=0 2015-06-17 11:09:13,044 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=1 queue=1 2015-06-17 11:09:13,045 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=2 queue=0 2015-06-17 11:09:13,045 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=3 queue=1 2015-06-17 11:09:13,045 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=4 queue=0 2015-06-17 11:09:13,045 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=5 queue=1 2015-06-17 11:09:13,045 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=6 queue=0 2015-06-17 11:09:13,045 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=7 queue=1 2015-06-17 11:09:13,045 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=8 queue=0 2015-06-17 11:09:13,045 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=9 queue=1 2015-06-17 11:09:13,045 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=10 queue=0 2015-06-17 11:09:13,046 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=11 queue=1 2015-06-17 11:09:13,046 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=12 queue=0 2015-06-17 11:09:13,046 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=13 queue=1 2015-06-17 11:09:13,046 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=14 queue=0 2015-06-17 11:09:13,046 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=15 queue=1 2015-06-17 11:09:13,046 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=16 queue=0 2015-06-17 11:09:13,046 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=17 queue=1 2015-06-17 11:09:13,046 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=18 queue=0 2015-06-17 11:09:13,046 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=19 queue=1 2015-06-17 11:09:13,047 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=0 queue=0 2015-06-17 11:09:13,047 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=1 queue=0 2015-06-17 11:09:13,047 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=2 queue=0 2015-06-17 11:09:13,047 INFO [MasterRestartThread] master.HMaster(349): hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034, hbase.cluster.distributed=false 2015-06-17 11:09:13,047 INFO [MasterRestartThread] master.HMaster(1512): Adding backup master ZNode /hbase/backup-masters/10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,048 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,049 INFO [M:0;10.0.0.110:51168] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x461d4103 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:09:13,049 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:09:13,049 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:09:13,050 DEBUG [10.0.0.110:51168.activeMasterManager] zookeeper.ZKUtil(486): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:09:13,050 INFO [10.0.0.110:51168.activeMasterManager] master.ActiveMasterManager(170): Deleting ZNode for /hbase/backup-masters/10.0.0.110,51168,1434564553036 from backup master directory 2015-06-17 11:09:13,050 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(486): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:09:13,050 DEBUG [MasterRestartThread-EventThread] master.ActiveMasterManager(126): A master is now available 2015-06-17 11:09:13,051 DEBUG [M:0;10.0.0.110:51168-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x461d41030x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:09:13,051 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,051 WARN [10.0.0.110:51168.activeMasterManager] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:09:13,051 INFO [10.0.0.110:51168.activeMasterManager] master.ActiveMasterManager(179): Registered Active Master=10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,051 DEBUG [M:0;10.0.0.110:51168] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6afee926, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:09:13,052 DEBUG [M:0;10.0.0.110:51168] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:09:13,052 DEBUG [M:0;10.0.0.110:51168] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:09:13,052 DEBUG [M:0;10.0.0.110:51168-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x461d4103-0x14e02b3a2320037 connected 2015-06-17 11:09:13,053 INFO [M:0;10.0.0.110:51168] regionserver.HRegionServer(774): ClusterId : b7c4b21b-aab2-44b4-8562-2b61af22ad6c 2015-06-17 11:09:13,053 DEBUG [M:0;10.0.0.110:51168] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2015-06-17 11:09:13,054 DEBUG [M:0;10.0.0.110:51168] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:09:13,055 DEBUG [M:0;10.0.0.110:51168] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2015-06-17 11:09:13,055 DEBUG [M:0;10.0.0.110:51168] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2015-06-17 11:09:13,055 DEBUG [M:0;10.0.0.110:51168] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:09:13,056 DEBUG [M:0;10.0.0.110:51168] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2015-06-17 11:09:13,056 INFO [M:0;10.0.0.110:51168] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2015-06-17 11:09:13,056 INFO [M:0;10.0.0.110:51168] compactions.PressureAwareCompactionThroughputController(175): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2015-06-17 11:09:13,057 INFO [M:0;10.0.0.110:51168] regionserver.HRegionServer$CompactionChecker(1504): CompactionChecker runs every 1sec 2015-06-17 11:09:13,057 DEBUG [M:0;10.0.0.110:51168] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40a9201b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=10.0.0.110/10.0.0.110:0 2015-06-17 11:09:13,057 DEBUG [M:0;10.0.0.110:51168] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:09:13,057 DEBUG [M:0;10.0.0.110:51168] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:09:13,057 DEBUG [M:0;10.0.0.110:51168] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:M:0;10.0.0.110:51168 2015-06-17 11:09:13,058 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:09:13,058 DEBUG [M:0;10.0.0.110:51168] zookeeper.ZKUtil(486): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,058 INFO [M:0;10.0.0.110:51168] regionserver.RegionServerCoprocessorHost(65): System coprocessor loading is enabled 2015-06-17 11:09:13,058 INFO [M:0;10.0.0.110:51168] regionserver.RegionServerCoprocessorHost(66): Table coprocessor loading is enabled 2015-06-17 11:09:13,058 INFO [M:0;10.0.0.110:51168] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,51168,1434564553036 with port=51168, startcode=1434564553036 2015-06-17 11:09:13,058 DEBUG [M:0;10.0.0.110:51168] regionserver.HRegionServer(2295): Master is not running yet 2015-06-17 11:09:13,058 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,058 WARN [M:0;10.0.0.110:51168] regionserver.HRegionServer(900): reportForDuty failed; sleeping and then retrying. 2015-06-17 11:09:13,059 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:13,063 DEBUG [10.0.0.110:51168.activeMasterManager] util.FSTableDescriptors(807): Current tableInfoPath = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2015-06-17 11:09:13,065 DEBUG [10.0.0.110:51168.activeMasterManager] util.FSTableDescriptors(811): TableInfo already exists.. Skipping creation 2015-06-17 11:09:13,068 INFO [10.0.0.110:51168.activeMasterManager] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:09:13,068 DEBUG [10.0.0.110:51168.activeMasterManager] coordination.ZKSplitLogManagerCoordination(885): Distributed log replay=true 2015-06-17 11:09:13,068 INFO [10.0.0.110:51168.activeMasterManager] coordination.ZKSplitLogManagerCoordination(599): Found 0 orphan tasks and 0 rescan nodes 2015-06-17 11:09:13,068 DEBUG [10.0.0.110:51168.activeMasterManager] util.FSTableDescriptors(222): Fetching table descriptors from the filesystem. 2015-06-17 11:09:13,078 INFO [10.0.0.110:51168.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x94c5391 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:09:13,080 DEBUG [10.0.0.110:51168.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x94c53910x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:09:13,080 DEBUG [10.0.0.110:51168.activeMasterManager] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@75d84668, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:09:13,080 DEBUG [10.0.0.110:51168.activeMasterManager] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:09:13,081 DEBUG [10.0.0.110:51168.activeMasterManager] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:09:13,081 DEBUG [10.0.0.110:51168.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x94c5391-0x14e02b3a2320038 connected 2015-06-17 11:09:13,081 INFO [10.0.0.110:51168.activeMasterManager] balancer.StochasticLoadBalancer(135): loading config 2015-06-17 11:09:13,082 DEBUG [10.0.0.110:51168.activeMasterManager] zookeeper.ZKUtil(488): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2015-06-17 11:09:13,082 DEBUG [10.0.0.110:51168.activeMasterManager] zookeeper.ZKUtil(486): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,082 DEBUG [10.0.0.110:51168.activeMasterManager] zookeeper.ZKUtil(486): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:13,083 DEBUG [10.0.0.110:51168.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,083 DEBUG [10.0.0.110:51168.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:13,083 INFO [10.0.0.110:51168.activeMasterManager] master.HMaster(568): Server active/primary master=10.0.0.110,51168,1434564553036, sessionid=0x14e02b3a2320036, setting cluster-up flag (Was=true) 2015-06-17 11:09:13,084 DEBUG [10.0.0.110:51168.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:09:13,085 INFO [10.0.0.110:51168.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2015-06-17 11:09:13,085 DEBUG [10.0.0.110:51168.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,086 DEBUG [10.0.0.110:51168.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:09:13,086 INFO [10.0.0.110:51168.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2015-06-17 11:09:13,087 DEBUG [10.0.0.110:51168.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,087 INFO [10.0.0.110:51168.activeMasterManager] master.MasterCoprocessorHost(86): System coprocessor loading is enabled 2015-06-17 11:09:13,087 DEBUG [10.0.0.110:51168.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_OPEN_REGION-10.0.0.110:51168, corePoolSize=5, maxPoolSize=5 2015-06-17 11:09:13,087 DEBUG [10.0.0.110:51168.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_CLOSE_REGION-10.0.0.110:51168, corePoolSize=5, maxPoolSize=5 2015-06-17 11:09:13,087 DEBUG [10.0.0.110:51168.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_SERVER_OPERATIONS-10.0.0.110:51168, corePoolSize=5, maxPoolSize=5 2015-06-17 11:09:13,087 DEBUG [10.0.0.110:51168.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_META_SERVER_OPERATIONS-10.0.0.110:51168, corePoolSize=5, maxPoolSize=5 2015-06-17 11:09:13,088 DEBUG [10.0.0.110:51168.activeMasterManager] executor.ExecutorService(91): Starting executor service name=M_LOG_REPLAY_OPS-10.0.0.110:51168, corePoolSize=10, maxPoolSize=10 2015-06-17 11:09:13,088 DEBUG [10.0.0.110:51168.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_TABLE_OPERATIONS-10.0.0.110:51168, corePoolSize=1, maxPoolSize=1 2015-06-17 11:09:13,088 INFO [10.0.0.110:51168.activeMasterManager] procedure2.ProcedureExecutor(446): Starting procedure executor threads=9 2015-06-17 11:09:13,088 INFO [10.0.0.110:51168.activeMasterManager] wal.WALProcedureStore(214): Starting WAL Procedure Store lease recovery 2015-06-17 11:09:13,089 INFO [10.0.0.110:51168.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log 2015-06-17 11:09:13,089 INFO [10.0.0.110:51168.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log after 0ms 2015-06-17 11:09:13,089 DEBUG [10.0.0.110:51168.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log; isDirectory=false; length=14915; replication=1; blocksize=134217728; modification_time=1434564413352; access_time=1434564405122; owner=ndimiduk; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:09:13,091 INFO [10.0.0.110:51168.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000010.log 2015-06-17 11:09:13,091 INFO [10.0.0.110:51168.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000010.log after 0ms 2015-06-17 11:09:13,091 DEBUG [10.0.0.110:51168.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000010.log; isDirectory=false; length=60; replication=1; blocksize=134217728; modification_time=1434564552481; access_time=1434564538980; owner=ndimiduk.hfs.9; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:09:13,095 DEBUG [10.0.0.110:51168.activeMasterManager] wal.WALProcedureStore(610): Roll new state log: 11 2015-06-17 11:09:13,096 INFO [10.0.0.110:51168.activeMasterManager] wal.WALProcedureStore(237): Lease acquired for flushLogId: 11 2015-06-17 11:09:13,096 INFO [10.0.0.110:51168.activeMasterManager] wal.ProcedureWALFormatReader(150): No active entry found in state log hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000010.log. removing it 2015-06-17 11:09:13,098 DEBUG [10.0.0.110:51168.activeMasterManager] procedure2.ProcedureExecutor$1(278): load procedures maxProcId=7 2015-06-17 11:09:13,099 DEBUG [10.0.0.110:51168.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=RUNNABLE isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=7 state=RUNNABLE 2015-06-17 11:09:13,099 DEBUG [10.0.0.110:51168.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DeleteTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=6 state=FINISHED 2015-06-17 11:09:13,099 DEBUG [10.0.0.110:51168.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:09:13,099 DEBUG [10.0.0.110:51168.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=5 state=FINISHED 2015-06-17 11:09:13,099 DEBUG [10.0.0.110:51168.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:09:13,099 DEBUG [10.0.0.110:51168.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: EnableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=4 state=FINISHED 2015-06-17 11:09:13,099 DEBUG [10.0.0.110:51168.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:09:13,099 DEBUG [10.0.0.110:51168.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=3 state=FINISHED 2015-06-17 11:09:13,099 DEBUG [10.0.0.110:51168.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:09:13,099 DEBUG [10.0.0.110:51168.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=2 state=FINISHED 2015-06-17 11:09:13,099 DEBUG [10.0.0.110:51168.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:09:13,099 DEBUG [10.0.0.110:51168.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=hbase:namespace) user=ndimiduk (auth:SIMPLE) id=1 state=FINISHED 2015-06-17 11:09:13,100 DEBUG [10.0.0.110:51168.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:09:13,100 DEBUG [10.0.0.110:51168.activeMasterManager] wal.WALProcedureStore(652): Remove log: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000010.log 2015-06-17 11:09:13,100 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=240, retries=350, started=60085 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:13,100 INFO [IPC Server handler 2 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741888_1064 127.0.0.1:50896 2015-06-17 11:09:13,102 DEBUG [10.0.0.110:51168.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2015-06-17 11:09:13,102 INFO [10.0.0.110:51168.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:09:13,104 DEBUG [10.0.0.110:51168.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:09:13,104 DEBUG [10.0.0.110:51168.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/replication/rs already exists 2015-06-17 11:09:13,104 DEBUG [10.0.0.110:51168.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2015-06-17 11:09:13,105 DEBUG [10.0.0.110:51168.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner 2015-06-17 11:09:13,105 DEBUG [10.0.0.110:51168.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): replicationLogCleaner-0x14e02b3a2320039 connected 2015-06-17 11:09:13,105 DEBUG [10.0.0.110:51168.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2015-06-17 11:09:13,106 DEBUG [10.0.0.110:51168.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2015-06-17 11:09:13,106 DEBUG [10.0.0.110:51168.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2015-06-17 11:09:13,106 INFO [10.0.0.110:51168.activeMasterManager] master.ServerManager(1011): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 1, timeout of 4500 ms, interval of 1500 ms. 2015-06-17 11:09:13,106 INFO [M:0;10.0.0.110:51168] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,51168,1434564553036 with port=51168, startcode=1434564553036 2015-06-17 11:09:13,106 INFO [M:0;10.0.0.110:51168] master.ServerManager(459): Registering server=10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,106 INFO [M:0;10.0.0.110:51168] regionserver.HRegionServer(1341): Config from master: hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034 2015-06-17 11:09:13,106 INFO [M:0;10.0.0.110:51168] regionserver.HRegionServer(1341): Config from master: fs.defaultFS=hdfs://localhost:50895 2015-06-17 11:09:13,106 INFO [M:0;10.0.0.110:51168] regionserver.HRegionServer(1341): Config from master: hbase.master.info.port=-1 2015-06-17 11:09:13,106 WARN [M:0;10.0.0.110:51168] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:09:13,106 INFO [M:0;10.0.0.110:51168] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:09:13,106 DEBUG [M:0;10.0.0.110:51168] regionserver.HRegionServer(1603): logdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,109 DEBUG [M:0;10.0.0.110:51168] regionserver.Replication(142): ReplicationStatisticsThread 300 2015-06-17 11:09:13,109 INFO [M:0;10.0.0.110:51168] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:09:13,111 INFO [M:0;10.0.0.110:51168] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.0.0.110%2C51168%2C1434564553036.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51168,1434564553036, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:09:13,117 INFO [M:0;10.0.0.110:51168] wal.FSHLog(1365): Slow sync cost: 3 ms, current pipeline: [] 2015-06-17 11:09:13,117 INFO [M:0;10.0.0.110:51168] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51168,1434564553036/10.0.0.110%2C51168%2C1434564553036.default.1434564553111 2015-06-17 11:09:13,117 INFO [M:0;10.0.0.110:51168] regionserver.MetricsRegionServerWrapperImpl(113): Computing regionserver metrics every 5000 milliseconds 2015-06-17 11:09:13,118 DEBUG [M:0;10.0.0.110:51168] executor.ExecutorService(91): Starting executor service name=RS_OPEN_REGION-10.0.0.110:51168, corePoolSize=3, maxPoolSize=3 2015-06-17 11:09:13,118 DEBUG [M:0;10.0.0.110:51168] executor.ExecutorService(91): Starting executor service name=RS_OPEN_META-10.0.0.110:51168, corePoolSize=1, maxPoolSize=1 2015-06-17 11:09:13,118 DEBUG [M:0;10.0.0.110:51168] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_REGION-10.0.0.110:51168, corePoolSize=3, maxPoolSize=3 2015-06-17 11:09:13,118 DEBUG [M:0;10.0.0.110:51168] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_META-10.0.0.110:51168, corePoolSize=1, maxPoolSize=1 2015-06-17 11:09:13,118 DEBUG [M:0;10.0.0.110:51168] executor.ExecutorService(91): Starting executor service name=RS_LOG_REPLAY_OPS-10.0.0.110:51168, corePoolSize=2, maxPoolSize=2 2015-06-17 11:09:13,118 DEBUG [M:0;10.0.0.110:51168] executor.ExecutorService(91): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-10.0.0.110:51168, corePoolSize=3, maxPoolSize=3 2015-06-17 11:09:13,120 DEBUG [M:0;10.0.0.110:51168] zookeeper.ZKUtil(486): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,120 DEBUG [M:0;10.0.0.110:51168] zookeeper.ZKUtil(486): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:13,120 INFO [M:0;10.0.0.110:51168] regionserver.ReplicationSourceManager(229): Current list of replicators: [10.0.0.110,51168,1434564553036, 10.0.0.110,50905,1434564404764] other RSs: [10.0.0.110,51168,1434564553036, 10.0.0.110,50905,1434564404764] 2015-06-17 11:09:13,133 INFO [M:0;10.0.0.110:51168] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x5d659833 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:09:13,135 DEBUG [M:0;10.0.0.110:51168-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x5d6598330x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:09:13,136 DEBUG [M:0;10.0.0.110:51168] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodecWithTags@61a298f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:09:13,136 DEBUG [M:0;10.0.0.110:51168] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:09:13,136 DEBUG [M:0;10.0.0.110:51168] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:09:13,137 DEBUG [M:0;10.0.0.110:51168-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x5d659833-0x14e02b3a232003a connected 2015-06-17 11:09:13,148 INFO [M:0;10.0.0.110:51168] regionserver.HeapMemoryManager(190): Starting HeapMemoryTuner chore. 2015-06-17 11:09:13,148 INFO [SplitLogWorker-10.0.0.110:51168] regionserver.SplitLogWorker(134): SplitLogWorker 10.0.0.110,51168,1434564553036 starting 2015-06-17 11:09:13,149 INFO [M:0;10.0.0.110:51168] regionserver.HRegionServer(1363): Serving as 10.0.0.110,51168,1434564553036, RpcServer on 10.0.0.110/10.0.0.110:51168, sessionid=0x14e02b3a2320036 2015-06-17 11:09:13,149 DEBUG [M:0;10.0.0.110:51168] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2015-06-17 11:09:13,149 DEBUG [M:0;10.0.0.110:51168] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager 10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,149 DEBUG [M:0;10.0.0.110:51168] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,51168,1434564553036' 2015-06-17 11:09:13,149 DEBUG [M:0;10.0.0.110:51168] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2015-06-17 11:09:13,149 DEBUG [M:0;10.0.0.110:51168] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2015-06-17 11:09:13,149 DEBUG [M:0;10.0.0.110:51168] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2015-06-17 11:09:13,149 DEBUG [M:0;10.0.0.110:51168] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2015-06-17 11:09:13,149 DEBUG [M:0;10.0.0.110:51168] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager 10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,149 DEBUG [M:0;10.0.0.110:51168] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,51168,1434564553036' 2015-06-17 11:09:13,150 DEBUG [M:0;10.0.0.110:51168] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2015-06-17 11:09:13,150 DEBUG [M:0;10.0.0.110:51168] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2015-06-17 11:09:13,150 DEBUG [M:0;10.0.0.110:51168] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2015-06-17 11:09:13,150 INFO [M:0;10.0.0.110:51168] quotas.RegionServerQuotaManager(62): Quota support disabled 2015-06-17 11:09:13,156 INFO [10.0.0.110:51168.activeMasterManager] master.ServerManager(1028): Finished waiting for region servers count to settle; checked in 1, slept for 50 ms, expecting minimum of 1, maximum of 1, master is running 2015-06-17 11:09:13,156 INFO [10.0.0.110:51168.activeMasterManager] master.ServerManager(459): Registering server=10.0.0.110,50905,1434564404764 2015-06-17 11:09:13,156 INFO [10.0.0.110:51168.activeMasterManager] master.HMaster(665): Registered server found up in zk but who has not yet reported in: 10.0.0.110,50905,1434564404764 2015-06-17 11:09:13,158 INFO [10.0.0.110:51168.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50905,1434564404764 belongs to an existing region server 2015-06-17 11:09:13,161 INFO [10.0.0.110:51168.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51168,1434564553036 belongs to an existing region server 2015-06-17 11:09:13,163 DEBUG [10.0.0.110:51168.activeMasterManager] coordination.ZKSplitLogManagerCoordination(650): Marked 1588230740 recovering from 10.0.0.110,51130,1434564521571: /hbase/recovering-regions/1588230740/10.0.0.110,51130,1434564521571 2015-06-17 11:09:13,163 INFO [10.0.0.110:51168.activeMasterManager] master.HMaster(884): Re-assigning hbase:meta with replicaId, 0 it was on 10.0.0.110,51130,1434564521571 2015-06-17 11:09:13,163 DEBUG [10.0.0.110:51168.activeMasterManager] master.AssignmentManager(1239): No previous transition plan found (or ignoring an existing plan) for hbase:meta,,1.1588230740; generated random plan=hri=hbase:meta,,1.1588230740, src=, dest=10.0.0.110,51168,1434564553036; 2 (online=2) available servers, forceNewPlan=false 2015-06-17 11:09:13,164 INFO [10.0.0.110:51168.activeMasterManager] master.AssignmentManager(1028): Assigning hbase:meta,,1.1588230740 to 10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,164 INFO [10.0.0.110:51168.activeMasterManager] master.RegionStates(1072): Transition {1588230740 state=OFFLINE, ts=1434564553163, server=10.0.0.110,51130,1434564521571} to {1588230740 state=PENDING_OPEN, ts=1434564553164, server=10.0.0.110,51168,1434564553036} 2015-06-17 11:09:13,164 INFO [10.0.0.110:51168.activeMasterManager] zookeeper.MetaTableLocator(440): Setting hbase:meta region location in ZooKeeper as 10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,164 DEBUG [10.0.0.110:51168.activeMasterManager] master.ServerManager(937): New admin connection to 10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,164 INFO [10.0.0.110:51168.activeMasterManager] regionserver.RSRpcServices(1432): Open hbase:meta,,1.1588230740 2015-06-17 11:09:13,165 DEBUG [10.0.0.110:51168.activeMasterManager] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564553165,"tag":[],"qualifier":"state","vlen":2}]},"row":"hbase:meta"} 2015-06-17 11:09:13,165 INFO [RS_OPEN_META-10.0.0.110:51168-0] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:09:13,168 INFO [RS_OPEN_META-10.0.0.110:51168-0] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.0.0.110%2C51168%2C1434564553036..meta, suffix=.meta, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51168,1434564553036, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:09:13,173 INFO [RS_OPEN_META-10.0.0.110:51168-0] wal.FSHLog(1365): Slow sync cost: 3 ms, current pipeline: [] 2015-06-17 11:09:13,174 INFO [RS_OPEN_META-10.0.0.110:51168-0] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51168,1434564553036/10.0.0.110%2C51168%2C1434564553036..meta.1434564553168.meta 2015-06-17 11:09:13,174 DEBUG [RS_OPEN_META-10.0.0.110:51168-0] regionserver.HRegion(6070): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2015-06-17 11:09:13,175 DEBUG [RS_OPEN_META-10.0.0.110:51168-0] coprocessor.CoprocessorHost(179): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2015-06-17 11:09:13,175 DEBUG [RS_OPEN_META-10.0.0.110:51168-0] regionserver.HRegion(7284): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2015-06-17 11:09:13,175 INFO [RS_OPEN_META-10.0.0.110:51168-0] regionserver.RegionCoprocessorHost(350): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2015-06-17 11:09:13,175 DEBUG [RS_OPEN_META-10.0.0.110:51168-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table meta 1588230740 2015-06-17 11:09:13,176 DEBUG [RS_OPEN_META-10.0.0.110:51168-0] regionserver.HRegion(722): Instantiated hbase:meta,,1.1588230740 2015-06-17 11:09:13,178 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:09:13,178 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:09:13,185 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/0cdbef2057bf48ffab39a58d41d9feda, isReference=false, isBulkLoadResult=false, seqid=120000147, majorCompaction=false 2015-06-17 11:09:13,190 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/b97cb9bad56b4097bc8516a91cd2d5c4, isReference=false, isBulkLoadResult=false, seqid=80000131, majorCompaction=true 2015-06-17 11:09:13,195 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/ebcc205b8b154961b65112b14f4a05ba, isReference=false, isBulkLoadResult=false, seqid=160000161, majorCompaction=false 2015-06-17 11:09:13,196 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:09:13,197 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:09:13,203 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/1b0c1561002046f7ac7b7f7dcc004d33, isReference=false, isBulkLoadResult=false, seqid=160000161, majorCompaction=false 2015-06-17 11:09:13,209 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/309e63f83f3a4850bf003589102a7393, isReference=false, isBulkLoadResult=false, seqid=80000131, majorCompaction=true 2015-06-17 11:09:13,215 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=81, retries=350, started=31260 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:13,215 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/9810d8ec635a40f199322a209fb0a8f6, isReference=false, isBulkLoadResult=false, seqid=120000147, majorCompaction=false 2015-06-17 11:09:13,217 DEBUG [RS_OPEN_META-10.0.0.110:51168-0] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740 2015-06-17 11:09:13,218 DEBUG [RS_OPEN_META-10.0.0.110:51168-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:09:13,221 DEBUG [RS_OPEN_META-10.0.0.110:51168-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/recovered.edits/200000164.seqid to file, newSeqId=200000164, maxSeqId=160000164 2015-06-17 11:09:13,222 INFO [RS_OPEN_META-10.0.0.110:51168-0] regionserver.HRegion(848): Onlined 1588230740; next sequenceid=200000164 2015-06-17 11:09:13,222 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1893): Post open deploy tasks for hbase:meta,,1.1588230740 2015-06-17 11:09:13,222 DEBUG [PostOpenDeployTasks:1588230740] regionserver.CompactSplitThread(356): Small Compaction requested: system; Because: Opening Region; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-06-17 11:09:13,222 DEBUG [PostOpenDeployTasks:1588230740] regionserver.CompactSplitThread(356): Small Compaction requested: system; Because: Opening Region; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-06-17 11:09:13,224 DEBUG [M:0;10.0.0.110:51168-longCompactions-1434564553056] compactions.RatioBasedCompactionPolicy(92): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 10 blocking 2015-06-17 11:09:13,224 DEBUG [M:0;10.0.0.110:51168-shortCompactions-1434564553222] compactions.RatioBasedCompactionPolicy(92): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 10 blocking 2015-06-17 11:09:13,225 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/recovering-regions/1588230740 2015-06-17 11:09:13,225 DEBUG [M:0;10.0.0.110:51168-longCompactions-1434564553056] compactions.ExploringCompactionPolicy(122): Exploring compaction algorithm has selected 3 files of size 14587 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-06-17 11:09:13,225 DEBUG [M:0;10.0.0.110:51168-shortCompactions-1434564553222] compactions.ExploringCompactionPolicy(122): Exploring compaction algorithm has selected 3 files of size 17552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-06-17 11:09:13,226 DEBUG [M:0;10.0.0.110:51168-longCompactions-1434564553056] regionserver.HStore(1568): 1588230740 - table: Initiating minor compaction (all files) 2015-06-17 11:09:13,226 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(3181): Update last flushed sequence id of region 1588230740 for 10.0.0.110,51130,1434564521571 2015-06-17 11:09:13,226 INFO [M:0;10.0.0.110:51168-longCompactions-1434564553056] regionserver.HRegion(1784): Starting compaction on table in region hbase:meta,,1.1588230740 2015-06-17 11:09:13,226 DEBUG [M:0;10.0.0.110:51168-shortCompactions-1434564553222] regionserver.HStore(1568): 1588230740 - info: Initiating minor compaction (all files) 2015-06-17 11:09:13,226 DEBUG [PostOpenDeployTasks:1588230740] master.AssignmentManager(2812): Got transition OPENED for {1588230740 state=PENDING_OPEN, ts=1434564553164, server=10.0.0.110,51168,1434564553036} from 10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,226 INFO [M:0;10.0.0.110:51168-shortCompactions-1434564553222] regionserver.HRegion(1784): Starting compaction on info in region hbase:meta,,1.1588230740 2015-06-17 11:09:13,226 INFO [M:0;10.0.0.110:51168-longCompactions-1434564553056] regionserver.HStore(1196): Starting compaction of 3 file(s) in table of hbase:meta,,1.1588230740 into tmpdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp, totalSize=14.2 K 2015-06-17 11:09:13,226 INFO [PostOpenDeployTasks:1588230740] master.RegionStates(1072): Transition {1588230740 state=PENDING_OPEN, ts=1434564553164, server=10.0.0.110,51168,1434564553036} to {1588230740 state=OPEN, ts=1434564553226, server=10.0.0.110,51168,1434564553036} 2015-06-17 11:09:13,226 INFO [PostOpenDeployTasks:1588230740] zookeeper.MetaTableLocator(440): Setting hbase:meta region location in ZooKeeper as 10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,226 INFO [M:0;10.0.0.110:51168-shortCompactions-1434564553222] regionserver.HStore(1196): Starting compaction of 3 file(s) in info of hbase:meta,,1.1588230740 into tmpdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp, totalSize=17.1 K 2015-06-17 11:09:13,227 DEBUG [M:0;10.0.0.110:51168-longCompactions-1434564553056] compactions.Compactor(169): Compacting hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/309e63f83f3a4850bf003589102a7393, keycount=8, bloomtype=NONE, size=5.0 K, encoding=NONE, seqNum=80000131, earliestPutTs=1434564405380 2015-06-17 11:09:13,227 DEBUG [PostOpenDeployTasks:1588230740] master.RegionStates(417): Onlined 1588230740 on 10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,227 DEBUG [M:0;10.0.0.110:51168-shortCompactions-1434564553222] compactions.Compactor(169): Compacting hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/b97cb9bad56b4097bc8516a91cd2d5c4, keycount=16, bloomtype=NONE, size=6.5 K, encoding=NONE, seqNum=80000131, earliestPutTs=1434564406293 2015-06-17 11:09:13,227 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1920): Finished post open deploy task for hbase:meta,,1.1588230740 2015-06-17 11:09:13,227 DEBUG [RS_OPEN_META-10.0.0.110:51168-0] handler.OpenRegionHandler(125): Opened hbase:meta,,1.1588230740 on 10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,228 DEBUG [M:0;10.0.0.110:51168-longCompactions-1434564553056] compactions.Compactor(169): Compacting hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/9810d8ec635a40f199322a209fb0a8f6, keycount=2, bloomtype=NONE, size=4.6 K, encoding=NONE, seqNum=120000147, earliestPutTs=1434564492756 2015-06-17 11:09:13,228 DEBUG [M:0;10.0.0.110:51168-shortCompactions-1434564553222] compactions.Compactor(169): Compacting hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/0cdbef2057bf48ffab39a58d41d9feda, keycount=6, bloomtype=NONE, size=5.3 K, encoding=NONE, seqNum=120000147, earliestPutTs=1434564492999 2015-06-17 11:09:13,228 DEBUG [M:0;10.0.0.110:51168-longCompactions-1434564553056] compactions.Compactor(169): Compacting hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/1b0c1561002046f7ac7b7f7dcc004d33, keycount=2, bloomtype=NONE, size=4.6 K, encoding=NONE, seqNum=160000161, earliestPutTs=1434564521700 2015-06-17 11:09:13,228 DEBUG [M:0;10.0.0.110:51168-shortCompactions-1434564553222] compactions.Compactor(169): Compacting hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/ebcc205b8b154961b65112b14f4a05ba, keycount=6, bloomtype=NONE, size=5.3 K, encoding=NONE, seqNum=160000161, earliestPutTs=1434564521940 2015-06-17 11:09:13,235 INFO [M:0;10.0.0.110:51168-shortCompactions-1434564553222] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:09:13,235 INFO [M:0;10.0.0.110:51168-longCompactions-1434564553056] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:09:13,238 INFO [M:0;10.0.0.110:51168-longCompactions-1434564553056] compactions.PressureAwareCompactionThroughputController(239): hbase:meta,,1.1588230740#table average throughput is 0.51 MB/sec, slept 0 time(s) and total slept time is 0 ms. 1 active compactions remaining, total limit is 10.00 MB/sec 2015-06-17 11:09:13,239 INFO [M:0;10.0.0.110:51168-shortCompactions-1434564553222] compactions.PressureAwareCompactionThroughputController(239): hbase:meta,,1.1588230740#info average throughput is 0.86 MB/sec, slept 0 time(s) and total slept time is 0 ms. 0 active compactions remaining, total limit is 10.00 MB/sec 2015-06-17 11:09:13,242 INFO [IPC Server handler 5 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741891_1067{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 5267 2015-06-17 11:09:13,243 INFO [IPC Server handler 6 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741892_1068{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 0 2015-06-17 11:09:13,244 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:09:13,246 DEBUG [AsyncRpcChannel-pool2-t5] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service MasterService, sasl=false 2015-06-17 11:09:13,246 DEBUG [RpcServer.listener,port=51168] ipc.RpcServer$Listener(840): RpcServer.listener,port=51168: connection from 10.0.0.110:51185; # active connections: 1 2015-06-17 11:09:13,247 INFO [RpcServer.reader=1,bindAddress=10.0.0.110,port=51168] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51185 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:09:13,248 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:13,248 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51168: callId: 1951 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:13,248 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4750 ms ago, cancelled=false, msg= 2015-06-17 11:09:13,251 DEBUG [M:0;10.0.0.110:51168-shortCompactions-1434564553222] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/cb290dc49abe4e33b88400230c798a8f as hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/cb290dc49abe4e33b88400230c798a8f 2015-06-17 11:09:13,257 DEBUG [M:0;10.0.0.110:51168-shortCompactions-1434564553222] regionserver.HStore(1703): Removing store files after compaction... 2015-06-17 11:09:13,258 DEBUG [M:0;10.0.0.110:51168-shortCompactions-1434564553222] backup.HFileArchiver(224): Archiving compacted store files. 2015-06-17 11:09:13,260 DEBUG [M:0;10.0.0.110:51168-shortCompactions-1434564553222] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/b97cb9bad56b4097bc8516a91cd2d5c4, to hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/archive/data/hbase/meta/1588230740/info/b97cb9bad56b4097bc8516a91cd2d5c4 2015-06-17 11:09:13,262 DEBUG [M:0;10.0.0.110:51168-shortCompactions-1434564553222] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/0cdbef2057bf48ffab39a58d41d9feda, to hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/archive/data/hbase/meta/1588230740/info/0cdbef2057bf48ffab39a58d41d9feda 2015-06-17 11:09:13,263 DEBUG [M:0;10.0.0.110:51168-shortCompactions-1434564553222] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/ebcc205b8b154961b65112b14f4a05ba, to hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/archive/data/hbase/meta/1588230740/info/ebcc205b8b154961b65112b14f4a05ba 2015-06-17 11:09:13,263 INFO [M:0;10.0.0.110:51168-shortCompactions-1434564553222] regionserver.HStore(1324): Completed compaction of 3 (all) file(s) in info of hbase:meta,,1.1588230740 into cb290dc49abe4e33b88400230c798a8f(size=6.5 K), total size for store is 6.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2015-06-17 11:09:13,263 INFO [M:0;10.0.0.110:51168-shortCompactions-1434564553222] regionserver.CompactSplitThread$CompactionRunner(527): Completed compaction: Request = regionName=hbase:meta,,1.1588230740, storeName=info, fileCount=3, fileSize=17.1 K, priority=7, time=1434564553226086000; duration=0sec 2015-06-17 11:09:13,263 DEBUG [M:0;10.0.0.110:51168-shortCompactions-1434564553222] regionserver.CompactSplitThread$CompactionRunner(550): CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-06-17 11:09:13,302 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=241, retries=350, started=60287 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:13,369 DEBUG [AsyncRpcChannel-pool2-t6] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service ClientService, sasl=false 2015-06-17 11:09:13,369 DEBUG [RpcServer.listener,port=51168] ipc.RpcServer$Listener(840): RpcServer.listener,port=51168: connection from 10.0.0.110:51187; # active connections: 2 2015-06-17 11:09:13,370 INFO [RpcServer.reader=2,bindAddress=10.0.0.110,port=51168] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51187 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:09:13,372 INFO [10.0.0.110:51168.activeMasterManager] hbase.MetaTableAccessor(1692): Updated table hbase:meta state to ENABLED in META 2015-06-17 11:09:13,374 DEBUG [10.0.0.110:51168.activeMasterManager] master.MasterFileSystem(339): Renamed region directory: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51130,1434564521571-splitting 2015-06-17 11:09:13,374 INFO [10.0.0.110:51168.activeMasterManager] master.SplitLogManager(560): dead splitlog workers [10.0.0.110,51130,1434564521571] 2015-06-17 11:09:13,375 INFO [10.0.0.110:51168.activeMasterManager] master.SplitLogManager(192): hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51130,1434564521571-splitting is empty dir, no logs to split 2015-06-17 11:09:13,375 DEBUG [10.0.0.110:51168.activeMasterManager] master.SplitLogManager(255): Scheduling batch of logs to split 2015-06-17 11:09:13,375 INFO [10.0.0.110:51168.activeMasterManager] master.SplitLogManager(257): started splitting 0 logs in [hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51130,1434564521571-splitting] for [10.0.0.110,51130,1434564521571] 2015-06-17 11:09:13,376 DEBUG [10.0.0.110:51168.activeMasterManager] coordination.ZKSplitLogManagerCoordination(324): Processing recovering [1588230740] and servers [10.0.0.110,51130,1434564521571], isMetaRecovery=true 2015-06-17 11:09:13,378 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/recovering-regions/1588230740/10.0.0.110,51130,1434564521571 2015-06-17 11:09:13,378 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/recovering-regions/1588230740 2015-06-17 11:09:13,380 INFO [RS_OPEN_REGION-10.0.0.110:51168-0] handler.FinishRegionRecoveringHandler(52): /hbase/recovering-regions/1588230740 deleted; 1588230740 recovered. 2015-06-17 11:09:13,380 INFO [10.0.0.110:51168.activeMasterManager] master.SplitLogManager(316): finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51130,1434564521571-splitting] in 5ms 2015-06-17 11:09:13,380 DEBUG [10.0.0.110:51168.activeMasterManager] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564553380,"tag":[],"qualifier":"state","vlen":2}]},"row":"hbase:meta"} 2015-06-17 11:09:13,383 INFO [10.0.0.110:51168.activeMasterManager] hbase.MetaTableAccessor(1692): Updated table hbase:meta state to ENABLED in META 2015-06-17 11:09:13,390 INFO [10.0.0.110:51168.activeMasterManager] master.ServerManager(686): AssignmentManager hasn't finished failover cleanup; waiting 2015-06-17 11:09:13,391 INFO [10.0.0.110:51168.activeMasterManager] master.HMaster(911): hbase:meta with replicaId 0 assigned=1, location=10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,408 INFO [10.0.0.110:51168.activeMasterManager] master.RegionStates(1072): Transition {9ccc7b1e8942a5472c9cf2d844c2fff0 state=OPEN, ts=1434564553404, server=10.0.0.110,51130,1434564521571} to {9ccc7b1e8942a5472c9cf2d844c2fff0 state=CLOSED, ts=1434564553408, server=10.0.0.110,51130,1434564521571} 2015-06-17 11:09:13,408 INFO [10.0.0.110:51168.activeMasterManager] master.AssignmentManager(538): Clean cluster startup. Assigning user regions 2015-06-17 11:09:13,408 INFO [10.0.0.110:51168.activeMasterManager] master.AssignmentManager(680): Assigning 1 region(s) to 10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,411 INFO [10.0.0.110:51168.activeMasterManager] master.RegionStates(1072): Transition {9ccc7b1e8942a5472c9cf2d844c2fff0 state=CLOSED, ts=1434564553408, server=10.0.0.110,51130,1434564521571} to {9ccc7b1e8942a5472c9cf2d844c2fff0 state=PENDING_OPEN, ts=1434564553411, server=10.0.0.110,51168,1434564553036} 2015-06-17 11:09:13,411 INFO [10.0.0.110:51168.activeMasterManager] master.RegionStateStore(207): Updating row hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. with state=PENDING_OPEN&sn=10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,412 INFO [10.0.0.110:51168.activeMasterManager] regionserver.RSRpcServices(1432): Open hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:09:13,413 DEBUG [10.0.0.110:51168.activeMasterManager] master.AssignmentManager(851): Bulk assigning done for 10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,413 DEBUG [RS_OPEN_REGION-10.0.0.110:51168-1] regionserver.HRegion(6070): Opening region: {ENCODED => 9ccc7b1e8942a5472c9cf2d844c2fff0, NAME => 'hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.', STARTKEY => '', ENDKEY => ''} 2015-06-17 11:09:13,414 DEBUG [RS_OPEN_REGION-10.0.0.110:51168-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table namespace 9ccc7b1e8942a5472c9cf2d844c2fff0 2015-06-17 11:09:13,414 DEBUG [RS_OPEN_REGION-10.0.0.110:51168-1] regionserver.HRegion(722): Instantiated hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:09:13,415 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=82, retries=350, started=31460 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:13,416 INFO [StoreOpener-9ccc7b1e8942a5472c9cf2d844c2fff0-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=4, currentSize=1076808, freeSize=1042885496, maxSize=1043962304, heapSize=1076808, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:09:13,416 INFO [StoreOpener-9ccc7b1e8942a5472c9cf2d844c2fff0-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:09:13,421 INFO [10.0.0.110:51168.activeMasterManager] master.AssignmentManager(420): Joined the cluster in 29ms, failover=false 2015-06-17 11:09:13,424 DEBUG [StoreOpener-9ccc7b1e8942a5472c9cf2d844c2fff0-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/info/b88c94ff75ab4fafba9b040819600c6e, isReference=false, isBulkLoadResult=false, seqid=8, majorCompaction=false 2015-06-17 11:09:13,425 DEBUG [10.0.0.110:51168.activeMasterManager] zookeeper.ZKUtil(486): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2015-06-17 11:09:13,426 DEBUG [10.0.0.110:51168.activeMasterManager] hbase.ZKNamespaceManager(196): Updating namespace cache from node default with data: \x0A\x07default 2015-06-17 11:09:13,426 DEBUG [10.0.0.110:51168.activeMasterManager] hbase.ZKNamespaceManager(196): Updating namespace cache from node hbase with data: \x0A\x05hbase 2015-06-17 11:09:13,426 DEBUG [RS_OPEN_REGION-10.0.0.110:51168-1] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0 2015-06-17 11:09:13,427 DEBUG [RS_OPEN_REGION-10.0.0.110:51168-1] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:09:13,431 DEBUG [RS_OPEN_REGION-10.0.0.110:51168-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/recovered.edits/24.seqid to file, newSeqId=24, maxSeqId=23 2015-06-17 11:09:13,431 INFO [RS_OPEN_REGION-10.0.0.110:51168-1] regionserver.HRegion(848): Onlined 9ccc7b1e8942a5472c9cf2d844c2fff0; next sequenceid=24 2015-06-17 11:09:13,432 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] regionserver.HRegionServer(1893): Post open deploy tasks for hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:09:13,433 DEBUG [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.AssignmentManager(2812): Got transition OPENED for {9ccc7b1e8942a5472c9cf2d844c2fff0 state=PENDING_OPEN, ts=1434564553411, server=10.0.0.110,51168,1434564553036} from 10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,433 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStates(1072): Transition {9ccc7b1e8942a5472c9cf2d844c2fff0 state=PENDING_OPEN, ts=1434564553411, server=10.0.0.110,51168,1434564553036} to {9ccc7b1e8942a5472c9cf2d844c2fff0 state=OPEN, ts=1434564553433, server=10.0.0.110,51168,1434564553036} 2015-06-17 11:09:13,433 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStateStore(207): Updating row hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. with state=OPEN&openSeqNum=24&server=10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,434 DEBUG [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStates(417): Onlined 9ccc7b1e8942a5472c9cf2d844c2fff0 on 10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,434 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStates(429): Offlined 9ccc7b1e8942a5472c9cf2d844c2fff0 from 10.0.0.110,51130,1434564521571 2015-06-17 11:09:13,435 DEBUG [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] regionserver.HRegionServer(1920): Finished post open deploy task for hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:09:13,435 DEBUG [RS_OPEN_REGION-10.0.0.110:51168-1] handler.OpenRegionHandler(125): Opened hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. on 10.0.0.110,51168,1434564553036 2015-06-17 11:09:13,449 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:13,449 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51168: callId: 1953 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:13,450 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4952 ms ago, cancelled=false, msg= 2015-06-17 11:09:13,503 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=242, retries=350, started=60488 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:13,615 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=83, retries=350, started=31660 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:13,648 DEBUG [M:0;10.0.0.110:51168-longCompactions-1434564553056] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/9a75bf859fde4bed8e2d6b731b927c46 as hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/9a75bf859fde4bed8e2d6b731b927c46 2015-06-17 11:09:13,651 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:13,651 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51168: callId: 1955 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:13,651 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5153 ms ago, cancelled=false, msg= 2015-06-17 11:09:13,655 DEBUG [M:0;10.0.0.110:51168-longCompactions-1434564553056] regionserver.HStore(1703): Removing store files after compaction... 2015-06-17 11:09:13,657 DEBUG [M:0;10.0.0.110:51168-longCompactions-1434564553056] backup.HFileArchiver(224): Archiving compacted store files. 2015-06-17 11:09:13,659 DEBUG [M:0;10.0.0.110:51168-longCompactions-1434564553056] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/309e63f83f3a4850bf003589102a7393, to hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/archive/data/hbase/meta/1588230740/table/309e63f83f3a4850bf003589102a7393 2015-06-17 11:09:13,660 DEBUG [M:0;10.0.0.110:51168-longCompactions-1434564553056] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/9810d8ec635a40f199322a209fb0a8f6, to hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/archive/data/hbase/meta/1588230740/table/9810d8ec635a40f199322a209fb0a8f6 2015-06-17 11:09:13,662 DEBUG [M:0;10.0.0.110:51168-longCompactions-1434564553056] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/1b0c1561002046f7ac7b7f7dcc004d33, to hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/archive/data/hbase/meta/1588230740/table/1b0c1561002046f7ac7b7f7dcc004d33 2015-06-17 11:09:13,663 INFO [M:0;10.0.0.110:51168-longCompactions-1434564553056] regionserver.HStore(1324): Completed compaction of 3 (all) file(s) in table of hbase:meta,,1.1588230740 into 9a75bf859fde4bed8e2d6b731b927c46(size=5.1 K), total size for store is 5.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2015-06-17 11:09:13,663 INFO [M:0;10.0.0.110:51168-longCompactions-1434564553056] regionserver.CompactSplitThread$CompactionRunner(527): Completed compaction: Request = regionName=hbase:meta,,1.1588230740, storeName=table, fileCount=3, fileSize=14.2 K, priority=7, time=1434564553226006000; duration=0sec 2015-06-17 11:09:13,663 DEBUG [M:0;10.0.0.110:51168-longCompactions-1434564553056] regionserver.CompactSplitThread$CompactionRunner(550): CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-06-17 11:09:13,705 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=243, retries=350, started=60690 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:13,816 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=84, retries=350, started=31861 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:13,854 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:13,854 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51168: callId: 1957 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:13,855 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5357 ms ago, cancelled=false, msg= 2015-06-17 11:09:13,855 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: Can't get master address from ZooKeeper; znode data == null Wed Jun 17 11:09:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564548498, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:09:13,907 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=244, retries=350, started=60892 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:14,018 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=85, retries=350, started=32063 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:14,057 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:14,057 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51168: callId: 1959 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:14,060 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:14,060 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51168: callId: 1961 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:14,064 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:14,064 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51168: callId: 1963 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:14,070 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:14,070 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51168: callId: 1965 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:14,081 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:14,081 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51168: callId: 1967 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:14,102 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:14,102 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51168: callId: 1969 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:14,108 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=245, retries=350, started=61093 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:14,143 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:14,143 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51168: callId: 1971 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:14,220 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=86, retries=350, started=32265 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:14,245 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:14,245 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51168: callId: 1973 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:14,308 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=246, retries=350, started=61293 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:14,346 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:14,347 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51168: callId: 1975 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:14,421 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=87, retries=350, started=32466 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:14,448 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:14,448 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51168: callId: 1977 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:14,509 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=247, retries=350, started=61494 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:14,550 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:14,550 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51168: callId: 1979 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:14,550 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=494 ms ago, cancelled=false, msg= 2015-06-17 11:09:14,579 DEBUG [AsyncRpcChannel-pool2-t8] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2015-06-17 11:09:14,579 DEBUG [RpcServer.listener,port=51168] ipc.RpcServer$Listener(840): RpcServer.listener,port=51168: connection from 10.0.0.110:51189; # active connections: 3 2015-06-17 11:09:14,580 INFO [RpcServer.reader=3,bindAddress=10.0.0.110,port=51168] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51189 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:09:14,621 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=88, retries=350, started=32666 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:14,710 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=248, retries=350, started=61694 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:14,752 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:14,753 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51168: callId: 1981 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:14,753 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=697 ms ago, cancelled=false, msg= 2015-06-17 11:09:14,821 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=89, retries=350, started=32866 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:14,911 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=249, retries=350, started=61896 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:14,954 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:14,955 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51168: callId: 1983 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:14,955 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=899 ms ago, cancelled=false, msg= 2015-06-17 11:09:15,023 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=90, retries=350, started=33068 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:15,111 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=250, retries=350, started=62096 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:15,156 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:15,156 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51168: callId: 1985 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:15,156 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1100 ms ago, cancelled=false, msg= 2015-06-17 11:09:15,224 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=91, retries=350, started=33269 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:15,311 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=251, retries=350, started=62296 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:15,359 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:15,360 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51168: callId: 1987 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:15,360 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1304 ms ago, cancelled=false, msg= 2015-06-17 11:09:15,425 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=92, retries=350, started=33470 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:15,513 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=252, retries=350, started=62498 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:15,563 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:15,564 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51168: callId: 1989 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:15,564 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1508 ms ago, cancelled=false, msg= 2015-06-17 11:09:15,626 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=93, retries=350, started=33671 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:15,715 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=253, retries=350, started=62700 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:15,765 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:15,765 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51168: callId: 1991 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:15,766 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1710 ms ago, cancelled=false, msg= 2015-06-17 11:09:15,827 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=94, retries=350, started=33872 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:15,917 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=254, retries=350, started=62902 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:15,968 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:15,968 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51168: callId: 1993 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:15,968 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1912 ms ago, cancelled=false, msg= 2015-06-17 11:09:16,029 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=95, retries=350, started=34073 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:16,117 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=255, retries=350, started=63102 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:16,171 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:16,171 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51168: callId: 1995 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:16,171 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2115 ms ago, cancelled=false, msg= 2015-06-17 11:09:16,231 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=96, retries=350, started=34276 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:16,317 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=256, retries=350, started=63302 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:16,375 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:16,376 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51168: callId: 1997 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:16,376 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2320 ms ago, cancelled=false, msg= 2015-06-17 11:09:16,431 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=97, retries=350, started=34476 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:16,519 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=257, retries=350, started=63504 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:16,577 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:16,577 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51168: callId: 1999 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:16,578 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2522 ms ago, cancelled=false, msg= 2015-06-17 11:09:16,632 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=98, retries=350, started=34677 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:16,720 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=258, retries=350, started=63705 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:16,779 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:16,779 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51168: callId: 2001 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:16,780 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2724 ms ago, cancelled=false, msg= 2015-06-17 11:09:16,834 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=99, retries=350, started=34879 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:16,920 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=259, retries=350, started=63905 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:16,982 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:16,982 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51168: callId: 2003 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:16,982 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2926 ms ago, cancelled=false, msg= 2015-06-17 11:09:17,036 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=100, retries=350, started=35081 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:17,121 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=260, retries=350, started=64106 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:17,185 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:17,185 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51168: callId: 2005 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:17,185 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3129 ms ago, cancelled=false, msg= 2015-06-17 11:09:17,238 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=101, retries=350, started=35283 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:17,322 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=261, retries=350, started=64307 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:17,386 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:17,386 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51168: callId: 2007 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:17,387 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3331 ms ago, cancelled=false, msg= 2015-06-17 11:09:17,439 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=102, retries=350, started=35484 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:17,522 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=262, retries=350, started=64507 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:17,588 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:17,588 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51168: callId: 2009 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:17,588 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3532 ms ago, cancelled=false, msg= 2015-06-17 11:09:17,641 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=103, retries=350, started=35686 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:17,722 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=263, retries=350, started=64707 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:17,789 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:17,789 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51168: callId: 2011 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:17,789 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3733 ms ago, cancelled=false, msg= 2015-06-17 11:09:17,843 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=104, retries=350, started=35888 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:17,925 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=264, retries=350, started=64910 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:17,991 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:17,992 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51168: callId: 2013 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:17,992 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3936 ms ago, cancelled=false, msg= 2015-06-17 11:09:18,043 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=105, retries=350, started=36088 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:18,125 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=265, retries=350, started=65110 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:18,195 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:18,195 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51168: callId: 2015 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:18,195 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4139 ms ago, cancelled=false, msg= 2015-06-17 11:09:18,244 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=106, retries=350, started=36289 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:18,327 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=266, retries=350, started=65312 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:18,396 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:18,397 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51168: callId: 2017 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:18,397 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4341 ms ago, cancelled=false, msg= 2015-06-17 11:09:18,445 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=107, retries=350, started=36490 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:18,529 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=267, retries=350, started=65514 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:18,599 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:18,599 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51168: callId: 2019 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:18,599 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4543 ms ago, cancelled=false, msg= 2015-06-17 11:09:18,646 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=108, retries=350, started=36691 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:18,731 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=268, retries=350, started=65716 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:18,802 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:18,802 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51168: callId: 2021 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:18,803 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4747 ms ago, cancelled=false, msg= 2015-06-17 11:09:18,847 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=109, retries=350, started=36892 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:18,933 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=269, retries=350, started=65918 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:19,004 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:19,004 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51168: callId: 2023 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:19,004 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4948 ms ago, cancelled=false, msg= 2015-06-17 11:09:19,049 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=110, retries=350, started=37094 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:19,058 INFO [ReplicationExecutor-0] replication.ReplicationQueuesZKImpl(171): Atomically moving 10.0.0.110,50975,1434564429984's WALs to my queue 2015-06-17 11:09:19,134 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=270, retries=350, started=66119 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:19,206 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:19,206 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51168: callId: 2025 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:19,207 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5151 ms ago, cancelled=false, msg= 2015-06-17 11:09:19,250 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=111, retries=350, started=37295 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:19,335 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=271, retries=350, started=66320 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:19,409 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:19,409 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51168: callId: 2027 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:19,409 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5353 ms ago, cancelled=false, msg= 2015-06-17 11:09:19,410 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564554056, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:09:19,451 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=112, retries=350, started=37496 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:19,536 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=272, retries=350, started=66521 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:19,611 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:19,611 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51168: callId: 2029 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:19,614 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:19,614 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51168: callId: 2031 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:19,618 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:19,619 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51168: callId: 2033 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:19,624 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:19,624 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51168: callId: 2035 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:19,635 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:19,635 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51168: callId: 2037 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:19,652 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=113, retries=350, started=37697 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:19,657 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:19,657 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51168: callId: 2039 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:19,699 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:19,700 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51168: callId: 2041 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:19,737 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=273, retries=350, started=66722 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:19,801 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:19,801 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51168: callId: 2043 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:19,854 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=114, retries=350, started=37899 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:19,903 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:19,903 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51168: callId: 2045 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:19,937 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=274, retries=350, started=66922 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:20,005 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:20,006 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51168: callId: 2047 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:20,055 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=115, retries=350, started=38100 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:20,108 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:20,108 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51168: callId: 2049 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:20,108 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=498 ms ago, cancelled=false, msg= 2015-06-17 11:09:20,139 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=275, retries=350, started=67124 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:20,257 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=116, retries=350, started=38302 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:20,311 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:20,311 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51168: callId: 2051 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:20,312 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=702 ms ago, cancelled=false, msg= 2015-06-17 11:09:20,339 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=276, retries=350, started=67324 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:20,457 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=117, retries=350, started=38502 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:20,513 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:20,513 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51168: callId: 2053 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:20,513 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=903 ms ago, cancelled=false, msg= 2015-06-17 11:09:20,539 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=277, retries=350, started=67524 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:20,658 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=118, retries=350, started=38703 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:20,715 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:20,715 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51168: callId: 2055 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:20,716 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1106 ms ago, cancelled=false, msg= 2015-06-17 11:09:20,741 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=278, retries=350, started=67726 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:20,860 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=119, retries=350, started=38905 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:20,917 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:20,917 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51168: callId: 2057 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:20,917 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1307 ms ago, cancelled=false, msg= 2015-06-17 11:09:20,941 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=279, retries=350, started=67926 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:21,060 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=120, retries=350, started=39105 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:21,120 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:21,120 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51168: callId: 2059 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:21,121 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1511 ms ago, cancelled=false, msg= 2015-06-17 11:09:21,143 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=280, retries=350, started=68128 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:21,262 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=121, retries=350, started=39307 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:21,324 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:21,324 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51168: callId: 2061 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:21,324 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1714 ms ago, cancelled=false, msg= 2015-06-17 11:09:21,344 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=281, retries=350, started=68329 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:21,464 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=122, retries=350, started=39509 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:21,526 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:21,526 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51168: callId: 2063 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:21,527 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1916 ms ago, cancelled=false, msg= 2015-06-17 11:09:21,545 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=282, retries=350, started=68530 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:21,664 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=123, retries=350, started=39709 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:21,727 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:21,728 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51168: callId: 2065 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:21,728 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2118 ms ago, cancelled=false, msg= 2015-06-17 11:09:21,746 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=283, retries=350, started=68731 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:21,865 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=124, retries=350, started=39910 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:21,931 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:21,931 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51168: callId: 2067 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:21,931 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2321 ms ago, cancelled=false, msg= 2015-06-17 11:09:21,948 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=284, retries=350, started=68933 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:22,066 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=125, retries=350, started=40111 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:22,133 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:22,134 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51168: callId: 2069 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:22,134 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2524 ms ago, cancelled=false, msg= 2015-06-17 11:09:22,148 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=285, retries=350, started=69133 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:22,268 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=126, retries=350, started=40313 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:22,336 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:22,336 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51168: callId: 2071 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:22,337 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2727 ms ago, cancelled=false, msg= 2015-06-17 11:09:22,348 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=286, retries=350, started=69333 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:22,468 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=127, retries=350, started=40513 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:22,538 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:22,538 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51168: callId: 2073 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:22,538 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2928 ms ago, cancelled=false, msg= 2015-06-17 11:09:22,550 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=287, retries=350, started=69535 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:22,670 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=128, retries=350, started=40715 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:22,740 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:22,741 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51168: callId: 2075 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:22,741 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3131 ms ago, cancelled=false, msg= 2015-06-17 11:09:22,751 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=288, retries=350, started=69736 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:22,871 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=129, retries=350, started=40916 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:22,944 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:22,944 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51168: callId: 2077 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:22,944 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3334 ms ago, cancelled=false, msg= 2015-06-17 11:09:22,952 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=289, retries=350, started=69937 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:23,072 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=130, retries=350, started=41117 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:23,079 DEBUG [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(188): Sleeping for 3500 before executing next master restart count=10 2015-06-17 11:09:23,147 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:23,147 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51168: callId: 2079 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:23,147 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3537 ms ago, cancelled=false, msg= 2015-06-17 11:09:23,154 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=290, retries=350, started=70139 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:23,274 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=131, retries=350, started=41319 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:23,349 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:23,349 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51168: callId: 2081 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:23,350 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3740 ms ago, cancelled=false, msg= 2015-06-17 11:09:23,354 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=291, retries=350, started=70339 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:23,475 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=132, retries=350, started=41520 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:23,552 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:23,552 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51168: callId: 2083 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:23,552 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3942 ms ago, cancelled=false, msg= 2015-06-17 11:09:23,555 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=292, retries=350, started=70540 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:23,678 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=133, retries=350, started=41723 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:23,755 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:23,755 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51168: callId: 2085 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:23,755 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4145 ms ago, cancelled=false, msg= 2015-06-17 11:09:23,758 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=293, retries=350, started=70743 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:23,879 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=134, retries=350, started=41924 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:23,957 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:23,957 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51168: callId: 2087 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:23,957 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4347 ms ago, cancelled=false, msg= 2015-06-17 11:09:23,958 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=294, retries=350, started=70943 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:24,079 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=135, retries=350, started=42124 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:24,158 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=295, retries=350, started=71143 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:24,158 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:24,158 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51168: callId: 2089 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:24,158 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4548 ms ago, cancelled=false, msg= 2015-06-17 11:09:24,281 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=136, retries=350, started=42326 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:24,359 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=296, retries=350, started=71344 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:24,360 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:24,361 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51168: callId: 2091 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:24,361 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4751 ms ago, cancelled=false, msg= 2015-06-17 11:09:24,482 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=137, retries=350, started=42527 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:24,559 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=297, retries=350, started=71544 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:24,562 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:24,562 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51168: callId: 2093 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:24,562 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4952 ms ago, cancelled=false, msg= 2015-06-17 11:09:24,682 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=138, retries=350, started=42727 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:24,761 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=298, retries=350, started=71746 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:24,764 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:24,764 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51168: callId: 2095 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:24,765 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5155 ms ago, cancelled=false, msg= 2015-06-17 11:09:24,883 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=139, retries=350, started=42928 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:24,963 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=299, retries=350, started=71948 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:24,966 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:24,966 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51168: callId: 2097 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:24,967 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5357 ms ago, cancelled=false, msg= 2015-06-17 11:09:24,967 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564559610, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:09:25,086 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=140, retries=350, started=43130 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:25,164 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=300, retries=350, started=72149 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:25,169 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:25,169 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51168: callId: 2099 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:25,173 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:25,173 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51168: callId: 2101 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:25,177 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:25,177 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51168: callId: 2103 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:25,183 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:25,183 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51168: callId: 2105 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:25,194 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:25,194 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51168: callId: 2107 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:25,215 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:25,216 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51168: callId: 2109 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:25,257 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:25,257 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51168: callId: 2111 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:25,287 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=141, retries=350, started=43332 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:25,360 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:25,360 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51168: callId: 2113 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:25,365 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=301, retries=350, started=72350 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:25,462 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:25,463 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51168: callId: 2115 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:25,489 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=142, retries=350, started=43534 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:25,564 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:25,565 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51168: callId: 2117 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:25,566 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=302, retries=350, started=72551 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:25,666 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:25,666 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51168: callId: 2119 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:25,667 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=498 ms ago, cancelled=false, msg= 2015-06-17 11:09:25,690 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=143, retries=350, started=43735 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:25,767 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=303, retries=350, started=72752 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:25,868 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:25,868 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51168: callId: 2121 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:25,869 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=700 ms ago, cancelled=false, msg= 2015-06-17 11:09:25,891 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=144, retries=350, started=43936 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:25,968 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=304, retries=350, started=72953 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:26,071 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:26,071 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51168: callId: 2123 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:26,071 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=902 ms ago, cancelled=false, msg= 2015-06-17 11:09:26,093 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=145, retries=350, started=44138 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:26,169 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=305, retries=350, started=73154 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:26,273 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:26,273 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51168: callId: 2125 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:26,274 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1105 ms ago, cancelled=false, msg= 2015-06-17 11:09:26,294 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=146, retries=350, started=44339 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:26,370 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=306, retries=350, started=73355 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:26,475 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:26,475 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51168: callId: 2127 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:26,475 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1306 ms ago, cancelled=false, msg= 2015-06-17 11:09:26,494 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=147, retries=350, started=44539 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:26,572 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=307, retries=350, started=73557 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:26,580 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(197): executing master kill 2015-06-17 11:09:26,580 INFO [MasterRestartThread] hbase.MiniHBaseCluster(422): Aborting 10.0.0.110,51168,1434564553036 2015-06-17 11:09:26,580 FATAL [MasterRestartThread] master.HMaster(1953): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2015-06-17 11:09:26,580 FATAL [MasterRestartThread] master.HMaster(1956): Aborting for tests java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:423) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:270) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$5.run(TestMasterRestartWithProcedures.java:198) 2015-06-17 11:09:26,580 INFO [MasterRestartThread] regionserver.HRegionServer(1859): STOPPED: Aborting for tests 2015-06-17 11:09:26,580 INFO [MasterRestartThread] hbase.LocalHBaseCluster(379): Waiting on 10.0.0.110,51168,1434564553036 2015-06-17 11:09:26,580 INFO [M:0;10.0.0.110:51168] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2015-06-17 11:09:26,580 INFO [M:0;10.0.0.110:51168] regionserver.HeapMemoryManager(201): Stoping HeapMemoryTuner chore. 2015-06-17 11:09:26,580 INFO [SplitLogWorker-10.0.0.110:51168] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2015-06-17 11:09:26,580 INFO [SplitLogWorker-10.0.0.110:51168] regionserver.SplitLogWorker(155): SplitLogWorker 10.0.0.110,51168,1434564553036 exiting 2015-06-17 11:09:26,581 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2015-06-17 11:09:26,580 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2015-06-17 11:09:26,580 INFO [M:0;10.0.0.110:51168] procedure2.ProcedureExecutor(490): Stopping the procedure executor 2015-06-17 11:09:26,582 INFO [M:0;10.0.0.110:51168] wal.WALProcedureStore(159): Stopping the WAL Procedure Store 2015-06-17 11:09:26,588 INFO [IPC Server handler 1 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741893_1069{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 0 2015-06-17 11:09:26,588 INFO [M:0;10.0.0.110:51168] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2015-06-17 11:09:26,588 INFO [M:0;10.0.0.110:51168] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2015-06-17 11:09:26,589 INFO [M:0;10.0.0.110:51168] regionserver.HRegionServer(1021): stopping server 10.0.0.110,51168,1434564553036 2015-06-17 11:09:26,589 DEBUG [M:0;10.0.0.110:51168] zookeeper.MetaTableLocator(617): Stopping MetaTableLocator 2015-06-17 11:09:26,589 DEBUG [RS_CLOSE_REGION-10.0.0.110:51168-0] handler.CloseRegionHandler(90): Processing close of hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:09:26,589 DEBUG [RS_CLOSE_REGION-10.0.0.110:51168-0] regionserver.HRegion(1365): Closing hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.: disabling compactions & flushes 2015-06-17 11:09:26,589 DEBUG [RS_CLOSE_REGION-10.0.0.110:51168-0] regionserver.HRegion(1392): Updates disabled for region hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:09:26,590 INFO [M:0;10.0.0.110:51168] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320037 2015-06-17 11:09:26,590 INFO [StoreCloserThread-hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.-1] regionserver.HStore(870): Closed info 2015-06-17 11:09:26,591 DEBUG [M:0;10.0.0.110:51168] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:09:26,591 INFO [M:0;10.0.0.110:51168] regionserver.CompactSplitThread(394): Waiting for Split Thread to finish... 2015-06-17 11:09:26,592 INFO [M:0;10.0.0.110:51168] regionserver.CompactSplitThread(394): Waiting for Merge Thread to finish... 2015-06-17 11:09:26,592 INFO [M:0;10.0.0.110:51168] regionserver.CompactSplitThread(394): Waiting for Large Compaction Thread to finish... 2015-06-17 11:09:26,592 DEBUG [AsyncRpcChannel-pool2-t6] ipc.AsyncRpcChannel$8(574): IPC Client (-1154908556) connection to /10.0.0.110:51168 from ndimiduk.hfs.10: closed 2015-06-17 11:09:26,592 DEBUG [RpcServer.reader=2,bindAddress=10.0.0.110,port=51168] ipc.RpcServer$Listener(872): RpcServer.listener,port=51168: DISCONNECTING client 10.0.0.110:51187 because read count=-1. Number of active connections: 3 2015-06-17 11:09:26,592 INFO [M:0;10.0.0.110:51168] regionserver.CompactSplitThread(394): Waiting for Small Compaction Thread to finish... 2015-06-17 11:09:26,592 DEBUG [AsyncRpcChannel-pool2-t7] ipc.AsyncRpcChannel$8(574): IPC Client (1460746118) connection to /10.0.0.110:51130 from ndimiduk.hfs.10: closed 2015-06-17 11:09:26,593 INFO [M:0;10.0.0.110:51168] regionserver.HRegionServer(1243): Waiting on 2 regions to close 2015-06-17 11:09:26,593 DEBUG [M:0;10.0.0.110:51168] regionserver.HRegionServer(1247): {9ccc7b1e8942a5472c9cf2d844c2fff0=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., 1588230740=hbase:meta,,1.1588230740} 2015-06-17 11:09:26,593 DEBUG [RS_CLOSE_META-10.0.0.110:51168-0] handler.CloseRegionHandler(90): Processing close of hbase:meta,,1.1588230740 2015-06-17 11:09:26,593 DEBUG [RS_CLOSE_META-10.0.0.110:51168-0] regionserver.HRegion(1365): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2015-06-17 11:09:26,594 DEBUG [RS_CLOSE_META-10.0.0.110:51168-0] regionserver.HRegion(1392): Updates disabled for region hbase:meta,,1.1588230740 2015-06-17 11:09:26,594 INFO [RS_CLOSE_META-10.0.0.110:51168-0] regionserver.HRegion(2111): Flushing 2/2 column families, memstore=1.75 KB 2015-06-17 11:09:26,597 DEBUG [RS_CLOSE_REGION-10.0.0.110:51168-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/recovered.edits/26.seqid to file, newSeqId=26, maxSeqId=24 2015-06-17 11:09:26,598 INFO [RS_CLOSE_REGION-10.0.0.110:51168-0] regionserver.HRegion(1486): Closed hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:09:26,598 DEBUG [RS_CLOSE_REGION-10.0.0.110:51168-0] handler.CloseRegionHandler(122): Closed hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:09:26,602 INFO [IPC Server handler 8 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741894_1070{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 5462 2015-06-17 11:09:26,682 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:26,683 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51168: callId: 2129 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:26,683 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1514 ms ago, cancelled=false, msg= 2015-06-17 11:09:26,695 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=148, retries=350, started=44740 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:26,774 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=308, retries=350, started=73759 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:26,887 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:26,887 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51168: callId: 2131 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:26,888 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1719 ms ago, cancelled=false, msg= 2015-06-17 11:09:26,897 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=149, retries=350, started=44942 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:26,976 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=309, retries=350, started=73960 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:27,003 INFO [RS_CLOSE_META-10.0.0.110:51168-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=200000177, memsize=1.4 K, hasBloomFilter=false, into tmp file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/b8513b7ab4244f5bb99ff80882e23b4c 2015-06-17 11:09:27,017 INFO [IPC Server handler 0 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741895_1071{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 0 2015-06-17 11:09:27,018 INFO [RS_CLOSE_META-10.0.0.110:51168-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=200000177, memsize=352, hasBloomFilter=false, into tmp file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/ad4ce718ee024b848754720b7c44b2fa 2015-06-17 11:09:27,023 DEBUG [RS_CLOSE_META-10.0.0.110:51168-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/b8513b7ab4244f5bb99ff80882e23b4c as hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/b8513b7ab4244f5bb99ff80882e23b4c 2015-06-17 11:09:27,028 INFO [RS_CLOSE_META-10.0.0.110:51168-0] regionserver.HStore(963): Added hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/b8513b7ab4244f5bb99ff80882e23b4c, entries=6, sequenceid=200000177, filesize=5.3 K 2015-06-17 11:09:27,029 DEBUG [RS_CLOSE_META-10.0.0.110:51168-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/ad4ce718ee024b848754720b7c44b2fa as hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/ad4ce718ee024b848754720b7c44b2fa 2015-06-17 11:09:27,035 INFO [RS_CLOSE_META-10.0.0.110:51168-0] regionserver.HStore(963): Added hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/ad4ce718ee024b848754720b7c44b2fa, entries=2, sequenceid=200000177, filesize=4.6 K 2015-06-17 11:09:27,036 INFO [RS_CLOSE_META-10.0.0.110:51168-0] regionserver.HRegion(2400): Finished memstore flush of ~1.75 KB/1792, currentsize=0 B/0 for region hbase:meta,,1.1588230740 in 442ms, sequenceid=200000177, compaction requested=false 2015-06-17 11:09:27,038 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(870): Closed info 2015-06-17 11:09:27,038 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(870): Closed table 2015-06-17 11:09:27,042 DEBUG [RS_CLOSE_META-10.0.0.110:51168-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/recovered.edits/200000180.seqid to file, newSeqId=200000180, maxSeqId=200000164 2015-06-17 11:09:27,043 DEBUG [RS_CLOSE_META-10.0.0.110:51168-0] coprocessor.CoprocessorHost(269): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2015-06-17 11:09:27,043 INFO [RS_CLOSE_META-10.0.0.110:51168-0] regionserver.HRegion(1486): Closed hbase:meta,,1.1588230740 2015-06-17 11:09:27,043 DEBUG [RS_CLOSE_META-10.0.0.110:51168-0] handler.CloseRegionHandler(122): Closed hbase:meta,,1.1588230740 2015-06-17 11:09:27,069 INFO [10.0.0.110,51168,1434564553036_splitLogManager__ChoreService_1] hbase.ScheduledChore(179): Chore: SplitLogManager Timeout Monitor was stopped 2015-06-17 11:09:27,092 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:27,092 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51168: callId: 2133 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:27,092 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1923 ms ago, cancelled=false, msg= 2015-06-17 11:09:27,099 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=150, retries=350, started=45144 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:27,119 INFO [10.0.0.110,51168,1434564553036_ChoreService_1] hbase.ScheduledChore(179): Chore: 10.0.0.110,51168,1434564553036-MemstoreFlusherChore was stopped 2015-06-17 11:09:27,127 INFO [master//10.0.0.110:0.logRoller] regionserver.LogRoller(159): LogRoller exiting. 2015-06-17 11:09:27,127 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(146): master//10.0.0.110:0.leaseChecker closing leases 2015-06-17 11:09:27,127 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(149): master//10.0.0.110:0.leaseChecker closed leases 2015-06-17 11:09:27,174 INFO [RS_OPEN_META-10.0.0.110:51168-0-MetaLogRoller] regionserver.LogRoller(159): LogRoller exiting. 2015-06-17 11:09:27,178 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=310, retries=350, started=74163 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:27,196 INFO [M:0;10.0.0.110:51168] regionserver.HRegionServer(1049): stopping server 10.0.0.110,51168,1434564553036; all regions closed. 2015-06-17 11:09:27,197 DEBUG [M:0;10.0.0.110:51168] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51168,1434564553036 2015-06-17 11:09:27,200 INFO [IPC Server handler 0 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741890_1066{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 2492 2015-06-17 11:09:27,279 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=350, started=13853 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:27,296 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:27,297 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51168: callId: 2135 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:27,297 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2128 ms ago, cancelled=false, msg= 2015-06-17 11:09:27,300 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=151, retries=350, started=45345 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:27,380 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=311, retries=350, started=74365 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:27,480 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=350, started=14054 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:27,500 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:27,500 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51168: callId: 2137 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:27,500 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2331 ms ago, cancelled=false, msg= 2015-06-17 11:09:27,503 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=152, retries=350, started=45548 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:27,580 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=312, retries=350, started=74565 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:27,605 DEBUG [M:0;10.0.0.110:51168] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:09:27,605 INFO [M:0;10.0.0.110:51168] wal.FSHLog(977): Closed WAL: FSHLog 10.0.0.110%2C51168%2C1434564553036..meta:.meta(num 1434564553168) 2015-06-17 11:09:27,605 DEBUG [M:0;10.0.0.110:51168] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51168,1434564553036 2015-06-17 11:09:27,608 INFO [IPC Server handler 9 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741889_1065{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 679 2015-06-17 11:09:27,682 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=350, started=14255 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:27,704 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=153, retries=350, started=45749 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:27,704 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:27,704 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51168: callId: 2139 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:27,705 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2536 ms ago, cancelled=false, msg= 2015-06-17 11:09:27,781 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=313, retries=350, started=74765 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:27,882 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=350, started=14456 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:27,906 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=154, retries=350, started=45951 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:27,908 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:27,908 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51168] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51168: callId: 2141 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51185 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:27,909 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2740 ms ago, cancelled=false, msg= 2015-06-17 11:09:27,983 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=314, retries=350, started=74968 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:28,013 DEBUG [M:0;10.0.0.110:51168] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:09:28,013 INFO [M:0;10.0.0.110:51168] wal.FSHLog(977): Closed WAL: FSHLog 10.0.0.110%2C51168%2C1434564553036.default:(num 1434564553111) 2015-06-17 11:09:28,013 DEBUG [M:0;10.0.0.110:51168] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:09:28,013 INFO [M:0;10.0.0.110:51168] regionserver.Leases(146): M:0;10.0.0.110:51168 closing leases 2015-06-17 11:09:28,013 INFO [M:0;10.0.0.110:51168] regionserver.Leases(149): M:0;10.0.0.110:51168 closed leases 2015-06-17 11:09:28,013 INFO [M:0;10.0.0.110:51168] hbase.ChoreService(302): Chore service for: 10.0.0.110,51168,1434564553036 had [[ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-10.0.0.110:51168 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region 10.0.0.110,51168,1434564553036 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.0.0.110,51168,1434564553036-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.0.0.110,51168,1434564553036-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS]] on shutdown 2015-06-17 11:09:28,015 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/10.0.0.110,51168,1434564553036 2015-06-17 11:09:28,015 INFO [M:0;10.0.0.110:51168] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a232003a 2015-06-17 11:09:28,016 DEBUG [M:0;10.0.0.110:51168] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:09:28,016 DEBUG [M:0;10.0.0.110:51168] master.HMaster(1074): Stopping service threads 2015-06-17 11:09:28,017 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:09:28,017 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:09:28,017 INFO [M:0;10.0.0.110:51168] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320038 2015-06-17 11:09:28,018 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(488): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:09:28,018 DEBUG [main-EventThread] zookeeper.ZKUtil(488): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:09:28,018 DEBUG [M:0;10.0.0.110:51168] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:09:28,018 INFO [M:0;10.0.0.110:51168] hbase.ChoreService(302): Chore service for: 10.0.0.110,51168,1434564553036_splitLogManager_ had [] on shutdown 2015-06-17 11:09:28,018 INFO [M:0;10.0.0.110:51168] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2015-06-17 11:09:28,018 INFO [M:0;10.0.0.110:51168] ipc.RpcServer(2219): Stopping server on 51168 2015-06-17 11:09:28,019 INFO [RpcServer.listener,port=51168] ipc.RpcServer$Listener(776): RpcServer.listener,port=51168: stopping 2015-06-17 11:09:28,019 INFO [RpcServer.responder] ipc.RpcServer$Responder(1018): RpcServer.responder: stopped 2015-06-17 11:09:28,019 INFO [RpcServer.responder] ipc.RpcServer$Responder(921): RpcServer.responder: stopping 2015-06-17 11:09:28,020 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,51168,1434564553036 2015-06-17 11:09:28,020 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,51168,1434564553036 2015-06-17 11:09:28,020 INFO [MasterRestartThread-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [10.0.0.110,51168,1434564553036] 2015-06-17 11:09:28,020 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51168-0x14e02b3a2320036, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:09:28,020 INFO [M:0;10.0.0.110:51168] regionserver.HRegionServer(1093): stopping server 10.0.0.110,51168,1434564553036; zookeeper connection closed. 2015-06-17 11:09:28,021 INFO [M:0;10.0.0.110:51168] regionserver.HRegionServer(1096): M:0;10.0.0.110:51168 exiting 2015-06-17 11:09:28,021 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(200): executing master restart 2015-06-17 11:09:28,021 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:28,021 INFO [main-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(121): /hbase/rs/10.0.0.110,51168,1434564553036 znode expired, triggering replicatorRemoved event 2015-06-17 11:09:28,022 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:09:28,022 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:28,039 INFO [MasterRestartThread] client.ConnectionUtils(105): master//10.0.0.110:0 server-side HConnection retries=350 2015-06-17 11:09:28,039 INFO [MasterRestartThread] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2015-06-17 11:09:28,042 INFO [MasterRestartThread] ipc.RpcServer$Listener(598): master//10.0.0.110:0: started 10 reader(s). 2015-06-17 11:09:28,045 INFO [MasterRestartThread] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:09:28,046 INFO [MasterRestartThread] zookeeper.RecoverableZooKeeper(121): Process identifier=master:51197 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:09:28,047 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:511970x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:09:28,048 DEBUG [MasterRestartThread] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-17 11:09:28,049 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(578): master:51197-0x14e02b3a232003b connected 2015-06-17 11:09:28,050 DEBUG [MasterRestartThread] zookeeper.ZKUtil(488): master:51197-0x14e02b3a232003b, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:09:28,050 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:51197-0x14e02b3a232003b, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2015-06-17 11:09:28,050 INFO [RpcServer.responder] ipc.RpcServer$Responder(917): RpcServer.responder: starting 2015-06-17 11:09:28,051 INFO [RpcServer.listener,port=51197] ipc.RpcServer$Listener(729): RpcServer.listener,port=51197: starting 2015-06-17 11:09:28,051 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=0 queue=0 2015-06-17 11:09:28,051 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=1 queue=0 2015-06-17 11:09:28,051 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=2 queue=0 2015-06-17 11:09:28,051 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=3 queue=0 2015-06-17 11:09:28,051 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=4 queue=0 2015-06-17 11:09:28,051 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=0 queue=0 2015-06-17 11:09:28,052 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=1 queue=1 2015-06-17 11:09:28,052 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=2 queue=0 2015-06-17 11:09:28,052 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=3 queue=1 2015-06-17 11:09:28,052 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=4 queue=0 2015-06-17 11:09:28,052 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=5 queue=1 2015-06-17 11:09:28,053 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=6 queue=0 2015-06-17 11:09:28,053 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=7 queue=1 2015-06-17 11:09:28,053 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=8 queue=0 2015-06-17 11:09:28,053 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=9 queue=1 2015-06-17 11:09:28,053 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=10 queue=0 2015-06-17 11:09:28,053 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=11 queue=1 2015-06-17 11:09:28,053 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=12 queue=0 2015-06-17 11:09:28,054 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=13 queue=1 2015-06-17 11:09:28,054 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=14 queue=0 2015-06-17 11:09:28,054 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=15 queue=1 2015-06-17 11:09:28,054 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=16 queue=0 2015-06-17 11:09:28,054 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=17 queue=1 2015-06-17 11:09:28,054 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=18 queue=0 2015-06-17 11:09:28,054 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=19 queue=1 2015-06-17 11:09:28,054 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=0 queue=0 2015-06-17 11:09:28,054 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=1 queue=0 2015-06-17 11:09:28,055 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=2 queue=0 2015-06-17 11:09:28,055 INFO [MasterRestartThread] master.HMaster(349): hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034, hbase.cluster.distributed=false 2015-06-17 11:09:28,055 INFO [MasterRestartThread] master.HMaster(1512): Adding backup master ZNode /hbase/backup-masters/10.0.0.110,51197,1434564568043 2015-06-17 11:09:28,056 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:51197-0x14e02b3a232003b, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/10.0.0.110,51197,1434564568043 2015-06-17 11:09:28,057 INFO [M:0;10.0.0.110:51197] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x19bf4e90 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:09:28,057 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51197-0x14e02b3a232003b, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:09:28,057 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:09:28,058 DEBUG [10.0.0.110:51197.activeMasterManager] zookeeper.ZKUtil(486): master:51197-0x14e02b3a232003b, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:09:28,058 INFO [10.0.0.110:51197.activeMasterManager] master.ActiveMasterManager(170): Deleting ZNode for /hbase/backup-masters/10.0.0.110,51197,1434564568043 from backup master directory 2015-06-17 11:09:28,058 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(486): master:51197-0x14e02b3a232003b, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:09:28,058 DEBUG [MasterRestartThread-EventThread] master.ActiveMasterManager(126): A master is now available 2015-06-17 11:09:28,059 DEBUG [M:0;10.0.0.110:51197-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x19bf4e900x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:09:28,060 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51197-0x14e02b3a232003b, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/10.0.0.110,51197,1434564568043 2015-06-17 11:09:28,060 WARN [10.0.0.110:51197.activeMasterManager] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:09:28,060 INFO [10.0.0.110:51197.activeMasterManager] master.ActiveMasterManager(179): Registered Active Master=10.0.0.110,51197,1434564568043 2015-06-17 11:09:28,060 DEBUG [M:0;10.0.0.110:51197] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@64e32491, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:09:28,060 DEBUG [M:0;10.0.0.110:51197] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:09:28,060 DEBUG [M:0;10.0.0.110:51197] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:09:28,061 DEBUG [M:0;10.0.0.110:51197-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x19bf4e90-0x14e02b3a232003c connected 2015-06-17 11:09:28,061 INFO [M:0;10.0.0.110:51197] regionserver.HRegionServer(774): ClusterId : b7c4b21b-aab2-44b4-8562-2b61af22ad6c 2015-06-17 11:09:28,061 DEBUG [M:0;10.0.0.110:51197] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2015-06-17 11:09:28,062 DEBUG [M:0;10.0.0.110:51197] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:09:28,063 DEBUG [M:0;10.0.0.110:51197] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2015-06-17 11:09:28,063 DEBUG [M:0;10.0.0.110:51197] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2015-06-17 11:09:28,063 DEBUG [M:0;10.0.0.110:51197] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:09:28,064 DEBUG [M:0;10.0.0.110:51197] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2015-06-17 11:09:28,064 INFO [M:0;10.0.0.110:51197] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2015-06-17 11:09:28,064 INFO [M:0;10.0.0.110:51197] compactions.PressureAwareCompactionThroughputController(175): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2015-06-17 11:09:28,064 INFO [M:0;10.0.0.110:51197] regionserver.HRegionServer$CompactionChecker(1504): CompactionChecker runs every 1sec 2015-06-17 11:09:28,065 DEBUG [M:0;10.0.0.110:51197] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@195f0951, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=10.0.0.110/10.0.0.110:0 2015-06-17 11:09:28,065 DEBUG [M:0;10.0.0.110:51197] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:09:28,065 DEBUG [M:0;10.0.0.110:51197] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:09:28,065 DEBUG [M:0;10.0.0.110:51197] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:M:0;10.0.0.110:51197 2015-06-17 11:09:28,066 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:09:28,066 DEBUG [M:0;10.0.0.110:51197] zookeeper.ZKUtil(486): master:51197-0x14e02b3a232003b, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51197,1434564568043 2015-06-17 11:09:28,066 INFO [M:0;10.0.0.110:51197] regionserver.RegionServerCoprocessorHost(65): System coprocessor loading is enabled 2015-06-17 11:09:28,066 INFO [M:0;10.0.0.110:51197] regionserver.RegionServerCoprocessorHost(66): Table coprocessor loading is enabled 2015-06-17 11:09:28,066 INFO [M:0;10.0.0.110:51197] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,51197,1434564568043 with port=51197, startcode=1434564568043 2015-06-17 11:09:28,066 DEBUG [M:0;10.0.0.110:51197] regionserver.HRegionServer(2295): Master is not running yet 2015-06-17 11:09:28,066 WARN [M:0;10.0.0.110:51197] regionserver.HRegionServer(900): reportForDuty failed; sleeping and then retrying. 2015-06-17 11:09:28,066 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51197,1434564568043 2015-06-17 11:09:28,067 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:28,071 DEBUG [10.0.0.110:51197.activeMasterManager] util.FSTableDescriptors(807): Current tableInfoPath = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2015-06-17 11:09:28,074 DEBUG [10.0.0.110:51197.activeMasterManager] util.FSTableDescriptors(811): TableInfo already exists.. Skipping creation 2015-06-17 11:09:28,077 INFO [10.0.0.110:51197.activeMasterManager] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:09:28,078 DEBUG [10.0.0.110:51197.activeMasterManager] coordination.ZKSplitLogManagerCoordination(885): Distributed log replay=true 2015-06-17 11:09:28,078 INFO [10.0.0.110:51197.activeMasterManager] coordination.ZKSplitLogManagerCoordination(599): Found 0 orphan tasks and 0 rescan nodes 2015-06-17 11:09:28,079 DEBUG [10.0.0.110:51197.activeMasterManager] util.FSTableDescriptors(222): Fetching table descriptors from the filesystem. 2015-06-17 11:09:28,082 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=350, started=14656 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:28,088 INFO [10.0.0.110:51197.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x1b42c313 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:09:28,090 DEBUG [10.0.0.110:51197.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x1b42c3130x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:09:28,090 DEBUG [10.0.0.110:51197.activeMasterManager] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@24c7f887, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:09:28,090 DEBUG [10.0.0.110:51197.activeMasterManager] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:09:28,090 DEBUG [10.0.0.110:51197.activeMasterManager] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:09:28,091 DEBUG [10.0.0.110:51197.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x1b42c313-0x14e02b3a232003d connected 2015-06-17 11:09:28,092 INFO [10.0.0.110:51197.activeMasterManager] balancer.StochasticLoadBalancer(135): loading config 2015-06-17 11:09:28,092 DEBUG [10.0.0.110:51197.activeMasterManager] zookeeper.ZKUtil(488): master:51197-0x14e02b3a232003b, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2015-06-17 11:09:28,093 DEBUG [10.0.0.110:51197.activeMasterManager] zookeeper.ZKUtil(486): master:51197-0x14e02b3a232003b, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51197,1434564568043 2015-06-17 11:09:28,093 DEBUG [10.0.0.110:51197.activeMasterManager] zookeeper.ZKUtil(486): master:51197-0x14e02b3a232003b, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:28,094 DEBUG [10.0.0.110:51197.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,51197,1434564568043 2015-06-17 11:09:28,094 DEBUG [10.0.0.110:51197.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:28,095 INFO [10.0.0.110:51197.activeMasterManager] master.HMaster(568): Server active/primary master=10.0.0.110,51197,1434564568043, sessionid=0x14e02b3a232003b, setting cluster-up flag (Was=true) 2015-06-17 11:09:28,097 DEBUG [10.0.0.110:51197.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:09:28,097 INFO [10.0.0.110:51197.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2015-06-17 11:09:28,098 DEBUG [10.0.0.110:51197.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,51197,1434564568043 2015-06-17 11:09:28,099 DEBUG [10.0.0.110:51197.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:09:28,099 INFO [10.0.0.110:51197.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2015-06-17 11:09:28,100 DEBUG [10.0.0.110:51197.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,51197,1434564568043 2015-06-17 11:09:28,100 INFO [10.0.0.110:51197.activeMasterManager] master.MasterCoprocessorHost(86): System coprocessor loading is enabled 2015-06-17 11:09:28,100 DEBUG [10.0.0.110:51197.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_OPEN_REGION-10.0.0.110:51197, corePoolSize=5, maxPoolSize=5 2015-06-17 11:09:28,100 DEBUG [10.0.0.110:51197.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_CLOSE_REGION-10.0.0.110:51197, corePoolSize=5, maxPoolSize=5 2015-06-17 11:09:28,100 DEBUG [10.0.0.110:51197.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_SERVER_OPERATIONS-10.0.0.110:51197, corePoolSize=5, maxPoolSize=5 2015-06-17 11:09:28,100 DEBUG [10.0.0.110:51197.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_META_SERVER_OPERATIONS-10.0.0.110:51197, corePoolSize=5, maxPoolSize=5 2015-06-17 11:09:28,100 DEBUG [10.0.0.110:51197.activeMasterManager] executor.ExecutorService(91): Starting executor service name=M_LOG_REPLAY_OPS-10.0.0.110:51197, corePoolSize=10, maxPoolSize=10 2015-06-17 11:09:28,100 DEBUG [10.0.0.110:51197.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_TABLE_OPERATIONS-10.0.0.110:51197, corePoolSize=1, maxPoolSize=1 2015-06-17 11:09:28,101 INFO [10.0.0.110:51197.activeMasterManager] procedure2.ProcedureExecutor(446): Starting procedure executor threads=9 2015-06-17 11:09:28,101 INFO [10.0.0.110:51197.activeMasterManager] wal.WALProcedureStore(214): Starting WAL Procedure Store lease recovery 2015-06-17 11:09:28,101 INFO [10.0.0.110:51197.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log 2015-06-17 11:09:28,102 INFO [10.0.0.110:51197.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log after 1ms 2015-06-17 11:09:28,102 DEBUG [10.0.0.110:51197.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log; isDirectory=false; length=14915; replication=1; blocksize=134217728; modification_time=1434564413352; access_time=1434564405122; owner=ndimiduk; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:09:28,104 INFO [10.0.0.110:51197.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000011.log 2015-06-17 11:09:28,104 INFO [10.0.0.110:51197.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000011.log after 0ms 2015-06-17 11:09:28,104 DEBUG [10.0.0.110:51197.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000011.log; isDirectory=false; length=60; replication=1; blocksize=134217728; modification_time=1434564566588; access_time=1434564553094; owner=ndimiduk.hfs.10; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:09:28,107 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=155, retries=350, started=46152 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:28,108 DEBUG [10.0.0.110:51197.activeMasterManager] wal.WALProcedureStore(610): Roll new state log: 12 2015-06-17 11:09:28,109 INFO [10.0.0.110:51197.activeMasterManager] wal.WALProcedureStore(237): Lease acquired for flushLogId: 12 2015-06-17 11:09:28,109 INFO [10.0.0.110:51197.activeMasterManager] wal.ProcedureWALFormatReader(150): No active entry found in state log hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000011.log. removing it 2015-06-17 11:09:28,110 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:09:28,111 DEBUG [AsyncRpcChannel-pool2-t9] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service MasterService, sasl=false 2015-06-17 11:09:28,111 DEBUG [10.0.0.110:51197.activeMasterManager] procedure2.ProcedureExecutor$1(278): load procedures maxProcId=7 2015-06-17 11:09:28,111 DEBUG [RpcServer.listener,port=51197] ipc.RpcServer$Listener(840): RpcServer.listener,port=51197: connection from 10.0.0.110:51202; # active connections: 1 2015-06-17 11:09:28,111 INFO [RpcServer.reader=1,bindAddress=10.0.0.110,port=51197] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51202 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:09:28,112 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51197: callId: 2143 service: MasterService methodName: IsMasterRunning size: 24 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:28,112 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2943 ms ago, cancelled=false, msg= 2015-06-17 11:09:28,112 DEBUG [10.0.0.110:51197.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=RUNNABLE isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=7 state=RUNNABLE 2015-06-17 11:09:28,112 DEBUG [10.0.0.110:51197.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DeleteTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=6 state=FINISHED 2015-06-17 11:09:28,112 DEBUG [10.0.0.110:51197.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:09:28,112 DEBUG [10.0.0.110:51197.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=5 state=FINISHED 2015-06-17 11:09:28,112 DEBUG [10.0.0.110:51197.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:09:28,112 DEBUG [10.0.0.110:51197.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: EnableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=4 state=FINISHED 2015-06-17 11:09:28,112 DEBUG [10.0.0.110:51197.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:09:28,113 DEBUG [10.0.0.110:51197.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=3 state=FINISHED 2015-06-17 11:09:28,113 DEBUG [10.0.0.110:51197.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:09:28,113 DEBUG [10.0.0.110:51197.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=2 state=FINISHED 2015-06-17 11:09:28,113 DEBUG [10.0.0.110:51197.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:09:28,113 DEBUG [10.0.0.110:51197.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=hbase:namespace) user=ndimiduk (auth:SIMPLE) id=1 state=FINISHED 2015-06-17 11:09:28,113 DEBUG [10.0.0.110:51197.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:09:28,113 DEBUG [10.0.0.110:51197.activeMasterManager] wal.WALProcedureStore(652): Remove log: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000011.log 2015-06-17 11:09:28,114 INFO [IPC Server handler 6 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741893_1069 127.0.0.1:50896 2015-06-17 11:09:28,115 DEBUG [10.0.0.110:51197.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2015-06-17 11:09:28,115 INFO [10.0.0.110:51197.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:09:28,117 DEBUG [10.0.0.110:51197.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:09:28,117 DEBUG [10.0.0.110:51197.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/replication/rs already exists 2015-06-17 11:09:28,117 DEBUG [10.0.0.110:51197.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2015-06-17 11:09:28,118 DEBUG [10.0.0.110:51197.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner 2015-06-17 11:09:28,118 DEBUG [10.0.0.110:51197.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2015-06-17 11:09:28,118 DEBUG [10.0.0.110:51197.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): replicationLogCleaner-0x14e02b3a232003e connected 2015-06-17 11:09:28,118 DEBUG [10.0.0.110:51197.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2015-06-17 11:09:28,119 DEBUG [10.0.0.110:51197.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2015-06-17 11:09:28,119 INFO [10.0.0.110:51197.activeMasterManager] master.ServerManager(1011): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 1, timeout of 4500 ms, interval of 1500 ms. 2015-06-17 11:09:28,119 INFO [M:0;10.0.0.110:51197] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,51197,1434564568043 with port=51197, startcode=1434564568043 2015-06-17 11:09:28,119 INFO [M:0;10.0.0.110:51197] master.ServerManager(459): Registering server=10.0.0.110,51197,1434564568043 2015-06-17 11:09:28,119 INFO [M:0;10.0.0.110:51197] regionserver.HRegionServer(1341): Config from master: hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034 2015-06-17 11:09:28,119 INFO [M:0;10.0.0.110:51197] regionserver.HRegionServer(1341): Config from master: fs.defaultFS=hdfs://localhost:50895 2015-06-17 11:09:28,119 INFO [M:0;10.0.0.110:51197] regionserver.HRegionServer(1341): Config from master: hbase.master.info.port=-1 2015-06-17 11:09:28,119 WARN [M:0;10.0.0.110:51197] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:09:28,119 INFO [M:0;10.0.0.110:51197] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:09:28,119 DEBUG [M:0;10.0.0.110:51197] regionserver.HRegionServer(1603): logdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51197,1434564568043 2015-06-17 11:09:28,122 DEBUG [M:0;10.0.0.110:51197] regionserver.Replication(142): ReplicationStatisticsThread 300 2015-06-17 11:09:28,122 INFO [M:0;10.0.0.110:51197] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:09:28,124 INFO [M:0;10.0.0.110:51197] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.0.0.110%2C51197%2C1434564568043.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51197,1434564568043, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:09:28,129 INFO [M:0;10.0.0.110:51197] wal.FSHLog(1365): Slow sync cost: 2 ms, current pipeline: [] 2015-06-17 11:09:28,129 INFO [M:0;10.0.0.110:51197] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51197,1434564568043/10.0.0.110%2C51197%2C1434564568043.default.1434564568125 2015-06-17 11:09:28,130 INFO [M:0;10.0.0.110:51197] regionserver.MetricsRegionServerWrapperImpl(113): Computing regionserver metrics every 5000 milliseconds 2015-06-17 11:09:28,130 DEBUG [M:0;10.0.0.110:51197] executor.ExecutorService(91): Starting executor service name=RS_OPEN_REGION-10.0.0.110:51197, corePoolSize=3, maxPoolSize=3 2015-06-17 11:09:28,130 DEBUG [M:0;10.0.0.110:51197] executor.ExecutorService(91): Starting executor service name=RS_OPEN_META-10.0.0.110:51197, corePoolSize=1, maxPoolSize=1 2015-06-17 11:09:28,130 DEBUG [M:0;10.0.0.110:51197] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_REGION-10.0.0.110:51197, corePoolSize=3, maxPoolSize=3 2015-06-17 11:09:28,131 DEBUG [M:0;10.0.0.110:51197] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_META-10.0.0.110:51197, corePoolSize=1, maxPoolSize=1 2015-06-17 11:09:28,131 DEBUG [M:0;10.0.0.110:51197] executor.ExecutorService(91): Starting executor service name=RS_LOG_REPLAY_OPS-10.0.0.110:51197, corePoolSize=2, maxPoolSize=2 2015-06-17 11:09:28,131 DEBUG [M:0;10.0.0.110:51197] executor.ExecutorService(91): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-10.0.0.110:51197, corePoolSize=3, maxPoolSize=3 2015-06-17 11:09:28,132 DEBUG [M:0;10.0.0.110:51197] zookeeper.ZKUtil(486): master:51197-0x14e02b3a232003b, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51197,1434564568043 2015-06-17 11:09:28,132 DEBUG [M:0;10.0.0.110:51197] zookeeper.ZKUtil(486): master:51197-0x14e02b3a232003b, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:28,132 INFO [M:0;10.0.0.110:51197] regionserver.ReplicationSourceManager(229): Current list of replicators: [10.0.0.110,51197,1434564568043, 10.0.0.110,50905,1434564404764] other RSs: [10.0.0.110,51197,1434564568043, 10.0.0.110,50905,1434564404764] 2015-06-17 11:09:28,145 INFO [M:0;10.0.0.110:51197] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x5227b654 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:09:28,147 DEBUG [M:0;10.0.0.110:51197-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x5227b6540x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:09:28,147 DEBUG [M:0;10.0.0.110:51197] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodecWithTags@3a8fd670, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:09:28,147 DEBUG [M:0;10.0.0.110:51197] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:09:28,147 DEBUG [M:0;10.0.0.110:51197] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:09:28,148 DEBUG [M:0;10.0.0.110:51197-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x5227b654-0x14e02b3a232003f connected 2015-06-17 11:09:28,160 INFO [M:0;10.0.0.110:51197] regionserver.HeapMemoryManager(190): Starting HeapMemoryTuner chore. 2015-06-17 11:09:28,160 INFO [SplitLogWorker-10.0.0.110:51197] regionserver.SplitLogWorker(134): SplitLogWorker 10.0.0.110,51197,1434564568043 starting 2015-06-17 11:09:28,160 INFO [M:0;10.0.0.110:51197] regionserver.HRegionServer(1363): Serving as 10.0.0.110,51197,1434564568043, RpcServer on 10.0.0.110/10.0.0.110:51197, sessionid=0x14e02b3a232003b 2015-06-17 11:09:28,160 DEBUG [M:0;10.0.0.110:51197] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2015-06-17 11:09:28,160 DEBUG [M:0;10.0.0.110:51197] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager 10.0.0.110,51197,1434564568043 2015-06-17 11:09:28,160 DEBUG [M:0;10.0.0.110:51197] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,51197,1434564568043' 2015-06-17 11:09:28,160 DEBUG [M:0;10.0.0.110:51197] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2015-06-17 11:09:28,160 DEBUG [M:0;10.0.0.110:51197] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2015-06-17 11:09:28,161 DEBUG [M:0;10.0.0.110:51197] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2015-06-17 11:09:28,161 DEBUG [M:0;10.0.0.110:51197] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2015-06-17 11:09:28,161 DEBUG [M:0;10.0.0.110:51197] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager 10.0.0.110,51197,1434564568043 2015-06-17 11:09:28,161 DEBUG [M:0;10.0.0.110:51197] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,51197,1434564568043' 2015-06-17 11:09:28,161 DEBUG [M:0;10.0.0.110:51197] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2015-06-17 11:09:28,161 DEBUG [M:0;10.0.0.110:51197] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2015-06-17 11:09:28,161 DEBUG [M:0;10.0.0.110:51197] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2015-06-17 11:09:28,161 INFO [M:0;10.0.0.110:51197] quotas.RegionServerQuotaManager(62): Quota support disabled 2015-06-17 11:09:28,170 INFO [10.0.0.110:51197.activeMasterManager] master.ServerManager(1028): Finished waiting for region servers count to settle; checked in 1, slept for 51 ms, expecting minimum of 1, maximum of 1, master is running 2015-06-17 11:09:28,170 INFO [10.0.0.110:51197.activeMasterManager] master.ServerManager(459): Registering server=10.0.0.110,50905,1434564404764 2015-06-17 11:09:28,170 INFO [10.0.0.110:51197.activeMasterManager] master.HMaster(665): Registered server found up in zk but who has not yet reported in: 10.0.0.110,50905,1434564404764 2015-06-17 11:09:28,172 INFO [10.0.0.110:51197.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50905,1434564404764 belongs to an existing region server 2015-06-17 11:09:28,175 INFO [10.0.0.110:51197.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51197,1434564568043 belongs to an existing region server 2015-06-17 11:09:28,176 WARN [10.0.0.110:51197.activeMasterManager] master.RegionStates(339): Open region's last host null should be the same as the current one 10.0.0.110,51168,1434564553036, ignored the last and used the current one 2015-06-17 11:09:28,183 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=315, retries=350, started=75168 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:28,283 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=350, started=14857 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:28,308 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=156, retries=350, started=46353 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:28,313 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:09:28,315 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:28,315 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51197: callId: 2146 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:28,315 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3146 ms ago, cancelled=false, msg= 2015-06-17 11:09:28,384 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=316, retries=350, started=75369 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:28,484 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=350, started=15058 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:28,509 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=157, retries=350, started=46554 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:28,518 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:28,518 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51197: callId: 2148 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:28,519 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3350 ms ago, cancelled=false, msg= 2015-06-17 11:09:28,584 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=317, retries=350, started=75569 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:28,685 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=350, started=15259 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:28,711 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=158, retries=350, started=46756 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:28,721 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:28,721 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51197: callId: 2150 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:28,721 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3552 ms ago, cancelled=false, msg= 2015-06-17 11:09:28,785 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=318, retries=350, started=75770 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:28,885 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=350, started=15459 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:28,912 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=159, retries=350, started=46957 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:28,923 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:28,923 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51197: callId: 2152 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:28,924 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3755 ms ago, cancelled=false, msg= 2015-06-17 11:09:28,986 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=319, retries=350, started=75971 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:29,087 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=350, started=15661 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:29,113 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=160, retries=350, started=47158 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:29,125 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:29,125 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51197: callId: 2154 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:29,125 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3956 ms ago, cancelled=false, msg= 2015-06-17 11:09:29,186 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=320, retries=350, started=76171 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:29,289 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=350, started=15863 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:29,313 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=161, retries=350, started=47358 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:29,332 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:29,332 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51197: callId: 2156 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:29,332 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4163 ms ago, cancelled=false, msg= 2015-06-17 11:09:29,388 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=321, retries=350, started=76373 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:29,490 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=350, started=16064 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:29,513 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=162, retries=350, started=47558 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:29,534 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:29,534 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51197: callId: 2158 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:29,535 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4366 ms ago, cancelled=false, msg= 2015-06-17 11:09:29,589 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=322, retries=350, started=76574 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:29,600 DEBUG [AsyncRpcChannel-pool2-t11] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2015-06-17 11:09:29,600 DEBUG [RpcServer.listener,port=51197] ipc.RpcServer$Listener(840): RpcServer.listener,port=51197: connection from 10.0.0.110:51207; # active connections: 2 2015-06-17 11:09:29,601 INFO [RpcServer.reader=2,bindAddress=10.0.0.110,port=51197] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51207 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:09:29,691 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=350, started=16265 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:29,715 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=163, retries=350, started=47760 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:29,737 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:29,737 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51197: callId: 2160 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:29,737 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4568 ms ago, cancelled=false, msg= 2015-06-17 11:09:29,792 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=323, retries=350, started=76777 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:29,894 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=350, started=16468 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:29,916 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=164, retries=350, started=47961 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:29,939 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:29,939 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51197: callId: 2162 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:29,940 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4771 ms ago, cancelled=false, msg= 2015-06-17 11:09:29,992 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=324, retries=350, started=76977 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:30,095 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=350, started=16669 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:30,118 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=165, retries=350, started=48163 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:30,143 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:30,143 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51197: callId: 2164 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:30,143 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4974 ms ago, cancelled=false, msg= 2015-06-17 11:09:30,194 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=325, retries=350, started=77179 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:30,295 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=350, started=16869 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:30,319 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=166, retries=350, started=48364 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:30,346 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:30,346 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51197: callId: 2166 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:30,347 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5177 ms ago, cancelled=false, msg= 2015-06-17 11:09:30,395 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=326, retries=350, started=77380 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:30,497 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=350, started=17071 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:30,519 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=167, retries=350, started=48564 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:30,549 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:30,549 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51197: callId: 2168 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:30,549 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5380 ms ago, cancelled=false, msg= 2015-06-17 11:09:30,550 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2028) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:890) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49808) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564565169, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:09:30,596 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=327, retries=350, started=77581 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:30,697 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=350, started=17271 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:30,720 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=168, retries=350, started=48765 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:30,751 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:30,751 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51197: callId: 2170 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:30,754 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:30,754 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51197: callId: 2172 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:30,758 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:30,758 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51197: callId: 2174 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:30,765 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:30,765 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51197: callId: 2176 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:30,776 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:30,776 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51197: callId: 2178 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:30,797 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=328, retries=350, started=77782 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:30,797 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:30,797 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51197: callId: 2180 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:30,840 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:30,840 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51197: callId: 2182 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:30,899 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=350, started=17473 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:30,920 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=169, retries=350, started=48965 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:30,942 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:30,942 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51197: callId: 2184 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:30,998 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=329, retries=350, started=77983 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:31,043 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:31,044 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51197: callId: 2186 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:31,101 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=350, started=17675 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:31,121 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=170, retries=350, started=49166 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:31,146 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:31,146 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51197: callId: 2188 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:31,200 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=330, retries=350, started=78185 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:31,247 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:31,248 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51197: callId: 2190 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:31,248 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=498 ms ago, cancelled=false, msg= 2015-06-17 11:09:31,303 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=350, started=17877 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:31,321 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=171, retries=350, started=49366 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:31,400 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=331, retries=350, started=78385 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:31,451 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:31,451 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51197: callId: 2192 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:31,451 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=701 ms ago, cancelled=false, msg= 2015-06-17 11:09:31,503 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=350, started=18077 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:31,522 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=172, retries=350, started=49567 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:31,601 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=332, retries=350, started=78585 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:31,653 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:31,653 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51197: callId: 2194 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:31,654 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=904 ms ago, cancelled=false, msg= 2015-06-17 11:09:31,705 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=350, started=18279 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:31,722 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=173, retries=350, started=49767 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:31,802 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=333, retries=350, started=78787 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:31,855 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:31,855 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51197: callId: 2196 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:31,855 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1105 ms ago, cancelled=false, msg= 2015-06-17 11:09:31,905 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=350, started=18479 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:31,922 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=174, retries=350, started=49967 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:32,002 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=334, retries=350, started=78987 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:32,058 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:32,058 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51197: callId: 2198 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:32,058 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1308 ms ago, cancelled=false, msg= 2015-06-17 11:09:32,106 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=350, started=18680 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:32,124 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=175, retries=350, started=50169 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:32,203 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=335, retries=350, started=79188 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:32,260 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:32,260 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51197: callId: 2200 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:32,260 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1510 ms ago, cancelled=false, msg= 2015-06-17 11:09:32,307 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=35, retries=350, started=18881 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:32,326 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=176, retries=350, started=50371 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:32,403 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=336, retries=350, started=79388 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:32,461 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:32,462 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51197: callId: 2202 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:32,462 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1712 ms ago, cancelled=false, msg= 2015-06-17 11:09:32,509 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=36, retries=350, started=19083 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:32,528 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=177, retries=350, started=50573 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:32,603 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=337, retries=350, started=79588 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:32,663 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:32,663 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51197: callId: 2204 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:32,663 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1913 ms ago, cancelled=false, msg= 2015-06-17 11:09:32,709 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=37, retries=350, started=19283 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:32,729 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=178, retries=350, started=50774 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:32,804 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=338, retries=350, started=79789 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:32,865 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:32,865 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51197: callId: 2206 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:32,865 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2115 ms ago, cancelled=false, msg= 2015-06-17 11:09:32,910 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=38, retries=350, started=19484 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:32,929 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=179, retries=350, started=50974 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:33,006 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=339, retries=350, started=79991 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:33,067 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:33,067 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51197: callId: 2208 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:33,068 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2318 ms ago, cancelled=false, msg= 2015-06-17 11:09:33,111 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=39, retries=350, started=19685 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:33,130 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=180, retries=350, started=51175 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:33,206 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=340, retries=350, started=80191 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:33,269 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:33,270 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51197: callId: 2210 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:33,270 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2520 ms ago, cancelled=false, msg= 2015-06-17 11:09:33,311 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=40, retries=350, started=19885 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:33,332 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=181, retries=350, started=51377 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:33,408 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=341, retries=350, started=80393 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:33,471 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:33,471 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51197: callId: 2212 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:33,471 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2721 ms ago, cancelled=false, msg= 2015-06-17 11:09:33,513 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=41, retries=350, started=20087 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:33,534 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=182, retries=350, started=51579 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:33,608 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=342, retries=350, started=80593 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:33,673 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:33,673 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51197: callId: 2214 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:33,674 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2924 ms ago, cancelled=false, msg= 2015-06-17 11:09:33,715 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=42, retries=350, started=20289 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:33,735 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=183, retries=350, started=51780 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:33,808 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=343, retries=350, started=80793 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:33,876 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:33,876 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51197: callId: 2216 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:33,876 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3126 ms ago, cancelled=false, msg= 2015-06-17 11:09:33,916 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=43, retries=350, started=20490 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:33,936 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=184, retries=350, started=51981 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:34,008 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=344, retries=350, started=80993 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:34,078 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:34,078 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51197: callId: 2218 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:34,079 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3329 ms ago, cancelled=false, msg= 2015-06-17 11:09:34,116 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=44, retries=350, started=20690 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:34,136 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=185, retries=350, started=52181 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:34,209 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=345, retries=350, started=81194 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:34,282 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:34,282 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51197: callId: 2220 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:34,282 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3532 ms ago, cancelled=false, msg= 2015-06-17 11:09:34,317 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=45, retries=350, started=20891 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:34,337 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=186, retries=350, started=52382 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:34,411 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=346, retries=350, started=81396 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:34,484 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:34,484 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51197: callId: 2222 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:34,485 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3735 ms ago, cancelled=false, msg= 2015-06-17 11:09:34,518 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=46, retries=350, started=21092 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:34,539 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=187, retries=350, started=52584 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:34,611 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=347, retries=350, started=81596 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:34,686 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:34,686 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51197: callId: 2224 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:34,686 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3936 ms ago, cancelled=false, msg= 2015-06-17 11:09:34,719 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=47, retries=350, started=21293 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:34,740 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=188, retries=350, started=52785 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:34,813 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=348, retries=350, started=81798 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:34,888 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:34,888 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51197: callId: 2226 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:34,888 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4138 ms ago, cancelled=false, msg= 2015-06-17 11:09:34,920 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=48, retries=350, started=21494 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:34,940 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=189, retries=350, started=52985 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:35,014 INFO [10.0.0.110:51074.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=349, retries=350, started=81999 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51015,1434564463167, seqNum=15 2015-06-17 11:09:35,017 WARN [10.0.0.110:51074.activeMasterManager] master.TableNamespaceManager(292): Caught exception in initializing namespace table manager org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=350, exceptions: Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call id=32 on server /10.0.0.110:51015 aborted: connection is closing Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:721) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:687) at org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:137) at org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:268) at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:98) at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:919) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:749) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1542) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: hconnection-0x3ea86006 closed at org.apache.hadoop.hbase.client.ConnectionImplementation.getTableState(ConnectionImplementation.java:2235) at org.apache.hadoop.hbase.client.ConnectionImplementation.isTableDisabled(ConnectionImplementation.java:553) at org.apache.hadoop.hbase.client.ConnectionImplementation.relocateRegion(ConnectionImplementation.java:718) at org.apache.hadoop.hbase.client.ConnectionImplementation.relocateRegion(ConnectionImplementation.java:706) at org.apache.hadoop.hbase.client.ConnectionImplementation.getRegionLocation(ConnectionImplementation.java:531) at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:81) at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:79) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) ... 10 more 2015-06-17 11:09:35,018 FATAL [10.0.0.110:51074.activeMasterManager] master.HMaster$1(1546): Failed to become active master org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=350, exceptions: Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call id=32 on server /10.0.0.110:51015 aborted: connection is closing Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:42 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:43 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:44 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:45 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:46 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:47 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:48 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:49 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:50 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:51 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:52 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:53 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:54 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:55 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:56 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:57 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:58 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:08:59 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:00 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:01 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:02 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:03 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:04 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:05 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:06 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:07 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:08 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:09 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:10 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:11 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:12 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:13 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:14 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:15 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:16 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:17 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:18 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:19 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:20 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:21 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:22 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:23 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:24 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:25 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:26 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:27 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:28 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:29 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed Wed Jun 17 11:09:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564493015, pause=1, retries=350}, java.io.IOException: hconnection-0x3ea86006 closed at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:721) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:687) at org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:137) at org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:268) at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:98) at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:919) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:749) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1542) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: hconnection-0x3ea86006 closed at org.apache.hadoop.hbase.client.ConnectionImplementation.getTableState(ConnectionImplementation.java:2235) at org.apache.hadoop.hbase.client.ConnectionImplementation.isTableDisabled(ConnectionImplementation.java:553) at org.apache.hadoop.hbase.client.ConnectionImplementation.relocateRegion(ConnectionImplementation.java:718) at org.apache.hadoop.hbase.client.ConnectionImplementation.relocateRegion(ConnectionImplementation.java:706) at org.apache.hadoop.hbase.client.ConnectionImplementation.getRegionLocation(ConnectionImplementation.java:531) at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:81) at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:79) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) ... 10 more 2015-06-17 11:09:35,091 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:35,091 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51197: callId: 2228 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:35,092 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4342 ms ago, cancelled=false, msg= 2015-06-17 11:09:35,121 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=49, retries=350, started=21695 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:35,140 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=190, retries=350, started=53185 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:35,293 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:35,294 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51197: callId: 2230 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:35,294 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4544 ms ago, cancelled=false, msg= 2015-06-17 11:09:35,322 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=50, retries=350, started=21896 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:35,341 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=191, retries=350, started=53386 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:35,497 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:35,497 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51197: callId: 2232 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:35,497 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4747 ms ago, cancelled=false, msg= 2015-06-17 11:09:35,523 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=51, retries=350, started=22097 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:35,543 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=192, retries=350, started=53588 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:35,699 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:35,699 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51197: callId: 2234 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:35,699 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4949 ms ago, cancelled=false, msg= 2015-06-17 11:09:35,725 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=52, retries=350, started=22299 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:35,743 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=193, retries=350, started=53788 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:35,900 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:35,901 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51197: callId: 2236 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:35,901 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5151 ms ago, cancelled=false, msg= 2015-06-17 11:09:35,925 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=53, retries=350, started=22499 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:35,943 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=194, retries=350, started=53988 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:36,103 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:36,103 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51197: callId: 2238 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:36,103 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5353 ms ago, cancelled=false, msg= 2015-06-17 11:09:36,104 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:09:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:30 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:31 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:32 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:33 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:34 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:35 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564570750, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:09:36,126 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=54, retries=350, started=22700 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:36,145 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=195, retries=350, started=54190 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:36,307 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:36,307 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51197: callId: 2240 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:36,310 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:36,310 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51197: callId: 2242 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:36,314 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:36,314 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51197: callId: 2244 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:36,320 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:36,320 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51197: callId: 2246 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:36,327 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=55, retries=350, started=22901 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:36,332 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:36,332 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51197: callId: 2248 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:36,345 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=196, retries=350, started=54390 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:36,354 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:36,354 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51197: callId: 2250 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:36,396 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:36,396 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51197: callId: 2252 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:36,497 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:36,497 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51197: callId: 2254 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:36,527 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=56, retries=350, started=23101 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:36,546 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=197, retries=350, started=54591 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:36,599 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:36,599 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51197: callId: 2256 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:36,700 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:36,701 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51197: callId: 2258 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:36,727 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=57, retries=350, started=23301 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:36,746 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=198, retries=350, started=54791 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:36,803 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:36,803 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51197: callId: 2260 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:36,803 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=497 ms ago, cancelled=false, msg= 2015-06-17 11:09:36,929 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=58, retries=350, started=23503 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:36,948 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=199, retries=350, started=54993 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:37,006 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:37,006 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51197: callId: 2262 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:37,007 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=701 ms ago, cancelled=false, msg= 2015-06-17 11:09:37,129 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=59, retries=350, started=23703 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:37,148 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=200, retries=350, started=55193 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:37,210 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:37,210 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51197: callId: 2264 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:37,210 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=904 ms ago, cancelled=false, msg= 2015-06-17 11:09:37,330 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=60, retries=350, started=23904 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:37,349 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=201, retries=350, started=55394 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:37,412 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:37,412 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51197: callId: 2266 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:37,412 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1106 ms ago, cancelled=false, msg= 2015-06-17 11:09:37,531 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=61, retries=350, started=24104 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:37,551 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=202, retries=350, started=55596 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:37,614 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:37,614 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51197: callId: 2268 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:37,614 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1308 ms ago, cancelled=false, msg= 2015-06-17 11:09:37,731 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=62, retries=350, started=24305 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:37,751 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=203, retries=350, started=55796 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:37,816 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:37,816 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51197: callId: 2270 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:37,816 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1510 ms ago, cancelled=false, msg= 2015-06-17 11:09:37,932 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=63, retries=350, started=24506 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:37,952 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=204, retries=350, started=55997 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:38,018 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:38,019 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51197: callId: 2272 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:38,019 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1713 ms ago, cancelled=false, msg= 2015-06-17 11:09:38,103 DEBUG [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(188): Sleeping for 4826 before executing next master restart count=11 2015-06-17 11:09:38,133 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=64, retries=350, started=24707 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:38,153 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=205, retries=350, started=56198 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:38,220 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:38,221 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51197: callId: 2274 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:38,221 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1915 ms ago, cancelled=false, msg= 2015-06-17 11:09:38,334 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=65, retries=350, started=24908 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:38,354 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=206, retries=350, started=56399 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:38,422 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:38,422 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51197: callId: 2276 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:38,423 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2117 ms ago, cancelled=false, msg= 2015-06-17 11:09:38,535 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=66, retries=350, started=25109 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:38,556 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=207, retries=350, started=56601 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:38,626 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:38,626 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51197: callId: 2278 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:38,626 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2320 ms ago, cancelled=false, msg= 2015-06-17 11:09:38,735 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=67, retries=350, started=25309 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:38,756 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=208, retries=350, started=56801 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:38,827 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:38,827 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51197: callId: 2280 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:38,828 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2522 ms ago, cancelled=false, msg= 2015-06-17 11:09:38,936 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=68, retries=350, started=25510 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:38,957 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=209, retries=350, started=57002 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:39,029 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:39,030 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51197: callId: 2282 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:39,030 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2724 ms ago, cancelled=false, msg= 2015-06-17 11:09:39,136 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=69, retries=350, started=25710 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:39,158 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=210, retries=350, started=57203 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:39,232 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:39,232 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51197: callId: 2284 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:39,232 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2926 ms ago, cancelled=false, msg= 2015-06-17 11:09:39,336 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=70, retries=350, started=25910 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:39,359 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=211, retries=350, started=57404 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:39,434 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:39,434 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51197: callId: 2286 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:39,434 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3128 ms ago, cancelled=false, msg= 2015-06-17 11:09:39,538 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=71, retries=350, started=26112 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:39,560 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=212, retries=350, started=57605 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:39,636 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:39,636 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51197: callId: 2288 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:39,636 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3330 ms ago, cancelled=false, msg= 2015-06-17 11:09:39,739 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=72, retries=350, started=26313 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:39,762 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=213, retries=350, started=57807 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:39,837 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:39,837 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51197: callId: 2290 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:39,838 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3532 ms ago, cancelled=false, msg= 2015-06-17 11:09:39,940 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=73, retries=350, started=26514 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:39,963 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=214, retries=350, started=58008 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:40,040 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:40,040 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51197: callId: 2292 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:40,040 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3734 ms ago, cancelled=false, msg= 2015-06-17 11:09:40,140 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=74, retries=350, started=26714 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:40,165 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=215, retries=350, started=58210 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:40,243 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:40,243 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51197: callId: 2294 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:40,244 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3938 ms ago, cancelled=false, msg= 2015-06-17 11:09:40,340 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=75, retries=350, started=26914 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:40,366 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=216, retries=350, started=58411 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:40,447 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:40,447 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51197: callId: 2296 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:40,447 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4141 ms ago, cancelled=false, msg= 2015-06-17 11:09:40,540 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=76, retries=350, started=27114 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:40,566 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=217, retries=350, started=58611 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:40,649 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:40,649 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51197: callId: 2298 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:40,649 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4343 ms ago, cancelled=false, msg= 2015-06-17 11:09:40,741 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=77, retries=350, started=27315 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:40,768 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=218, retries=350, started=58813 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:40,850 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:40,850 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51197: callId: 2300 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:40,851 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4545 ms ago, cancelled=false, msg= 2015-06-17 11:09:40,942 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=78, retries=350, started=27516 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:40,970 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=219, retries=350, started=59015 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:41,053 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:41,054 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51197: callId: 2302 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:41,054 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4748 ms ago, cancelled=false, msg= 2015-06-17 11:09:41,143 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=79, retries=350, started=27717 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:41,172 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=220, retries=350, started=59217 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:41,257 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:41,257 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51197: callId: 2304 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:41,257 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=32, retries=35, started=4951 ms ago, cancelled=false, msg= 2015-06-17 11:09:41,344 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=80, retries=350, started=27918 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:41,373 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=221, retries=350, started=59418 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:41,459 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:41,459 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51197: callId: 2306 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:41,460 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=33, retries=35, started=5154 ms ago, cancelled=false, msg= 2015-06-17 11:09:41,545 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=81, retries=350, started=28119 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:41,575 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=222, retries=350, started=59620 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:41,662 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:41,662 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51197: callId: 2308 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:41,662 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=34, retries=35, started=5356 ms ago, cancelled=false, msg= 2015-06-17 11:09:41,662 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions: Wed Jun 17 11:09:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:36 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:37 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:38 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:39 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:40 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Wed Jun 17 11:09:41 PDT 2015, RpcRetryingCaller{globalStartTime=1434564576306, pause=1, retries=35}, org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:139) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.ipc.AsyncCall.setFailed(AsyncCall.java:122) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.createRemoteException(AsyncServerResponseHandler.java:120) at org.apache.hadoop.hbase.ipc.AsyncServerResponseHandler.channelRead(AsyncServerResponseHandler.java:81) ... 13 more 2015-06-17 11:09:41,747 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=82, retries=350, started=28321 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:41,776 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=223, retries=350, started=59821 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:41,865 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:41,865 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51197: callId: 2310 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:41,868 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:41,868 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51197: callId: 2312 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:41,873 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:41,873 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51197: callId: 2314 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:41,879 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:41,879 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51197: callId: 2316 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:41,891 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:41,891 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51197: callId: 2318 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:41,913 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:41,913 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51197: callId: 2320 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:41,948 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=83, retries=350, started=28522 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:41,955 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:41,955 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51197: callId: 2322 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:41,977 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=224, retries=350, started=60022 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:42,056 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:42,056 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51197: callId: 2324 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:42,149 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=84, retries=350, started=28723 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:42,159 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:42,159 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51197: callId: 2326 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:42,179 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=225, retries=350, started=60224 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:42,260 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:42,260 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51197: callId: 2328 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:42,350 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=85, retries=350, started=28924 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:42,361 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:42,362 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51197: callId: 2330 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:42,362 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=498 ms ago, cancelled=false, msg= 2015-06-17 11:09:42,380 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=226, retries=350, started=60425 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:42,551 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=86, retries=350, started=29125 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:42,563 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:42,563 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51197: callId: 2332 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:42,563 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=699 ms ago, cancelled=false, msg= 2015-06-17 11:09:42,580 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=227, retries=350, started=60625 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:42,751 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=87, retries=350, started=29325 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:42,765 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:42,765 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51197: callId: 2334 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:42,766 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=902 ms ago, cancelled=false, msg= 2015-06-17 11:09:42,781 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=228, retries=350, started=60826 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:42,929 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(197): executing master kill 2015-06-17 11:09:42,929 INFO [MasterRestartThread] hbase.MiniHBaseCluster(422): Aborting 10.0.0.110,51197,1434564568043 2015-06-17 11:09:42,929 FATAL [MasterRestartThread] master.HMaster(1953): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2015-06-17 11:09:42,929 FATAL [MasterRestartThread] master.HMaster(1956): Aborting for tests java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:423) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:270) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$5.run(TestMasterRestartWithProcedures.java:198) 2015-06-17 11:09:42,930 INFO [MasterRestartThread] regionserver.HRegionServer(1859): STOPPED: Aborting for tests 2015-06-17 11:09:42,930 INFO [MasterRestartThread] hbase.LocalHBaseCluster(379): Waiting on 10.0.0.110,51197,1434564568043 2015-06-17 11:09:42,930 INFO [M:0;10.0.0.110:51197] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2015-06-17 11:09:42,930 INFO [M:0;10.0.0.110:51197] regionserver.HeapMemoryManager(201): Stoping HeapMemoryTuner chore. 2015-06-17 11:09:42,930 INFO [SplitLogWorker-10.0.0.110:51197] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2015-06-17 11:09:42,930 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2015-06-17 11:09:42,930 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2015-06-17 11:09:42,930 INFO [SplitLogWorker-10.0.0.110:51197] regionserver.SplitLogWorker(155): SplitLogWorker 10.0.0.110,51197,1434564568043 exiting 2015-06-17 11:09:42,930 INFO [M:0;10.0.0.110:51197] procedure2.ProcedureExecutor(490): Stopping the procedure executor 2015-06-17 11:09:42,931 INFO [M:0;10.0.0.110:51197] wal.WALProcedureStore(159): Stopping the WAL Procedure Store 2015-06-17 11:09:42,937 INFO [IPC Server handler 5 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741897_1073{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 0 2015-06-17 11:09:42,938 INFO [M:0;10.0.0.110:51197] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2015-06-17 11:09:42,938 INFO [M:0;10.0.0.110:51197] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2015-06-17 11:09:42,938 INFO [M:0;10.0.0.110:51197] regionserver.HRegionServer(1021): stopping server 10.0.0.110,51197,1434564568043 2015-06-17 11:09:42,938 DEBUG [M:0;10.0.0.110:51197] zookeeper.MetaTableLocator(617): Stopping MetaTableLocator 2015-06-17 11:09:42,938 INFO [M:0;10.0.0.110:51197] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a232003c 2015-06-17 11:09:42,939 DEBUG [M:0;10.0.0.110:51197] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:09:42,939 INFO [M:0;10.0.0.110:51197] regionserver.HRegionServer(1049): stopping server 10.0.0.110,51197,1434564568043; all regions closed. 2015-06-17 11:09:42,939 DEBUG [M:0;10.0.0.110:51197] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51197,1434564568043 2015-06-17 11:09:42,939 INFO [10.0.0.110:51197.activeMasterManager] zookeeper.MetaTableLocator(340): Failed verification of hbase:meta,,1 at address=10.0.0.110,51168,1434564553036, exception=Call id=0 on server /10.0.0.110:51168 aborted: connection is closing 2015-06-17 11:09:42,939 DEBUG [AsyncRpcChannel-pool2-t10] ipc.AsyncRpcChannel$8(574): IPC Client (-738730664) connection to /10.0.0.110:51168 from ndimiduk.hfs.11: closed 2015-06-17 11:09:42,940 DEBUG [10.0.0.110:51197.activeMasterManager] zookeeper.ZKUtil(745): master:51197-0x14e02b3a232003b, quorum=localhost:51848, baseZNode=/hbase Unable to get data of znode /hbase/recovering-regions/1588230740 because node does not exist (not an error) 2015-06-17 11:09:42,941 INFO [IPC Server handler 8 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741896_1072{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 91 2015-06-17 11:09:42,941 DEBUG [10.0.0.110:51197.activeMasterManager] coordination.ZKSplitLogManagerCoordination(650): Marked 1588230740 recovering from 10.0.0.110,51168,1434564553036: /hbase/recovering-regions/1588230740/10.0.0.110,51168,1434564553036 2015-06-17 11:09:42,941 INFO [10.0.0.110:51197.activeMasterManager] master.HMaster(884): Re-assigning hbase:meta with replicaId, 0 it was on 10.0.0.110,51168,1434564553036 2015-06-17 11:09:42,941 INFO [10.0.0.110:51197.activeMasterManager] master.RegionStates(1072): Transition {1588230740 state=OPEN, ts=1434564568176, server=10.0.0.110,51168,1434564553036} to {1588230740 state=OFFLINE, ts=1434564582941, server=10.0.0.110,51168,1434564553036} 2015-06-17 11:09:42,941 INFO [10.0.0.110:51197.activeMasterManager] zookeeper.MetaTableLocator(440): Setting hbase:meta region location in ZooKeeper as 10.0.0.110,51168,1434564553036 2015-06-17 11:09:42,942 INFO [10.0.0.110:51197.activeMasterManager] master.AssignmentManager(996): Skip assigning hbase:meta,,1.1588230740, the server is stopped/aborted 2015-06-17 11:09:42,942 FATAL [10.0.0.110:51197.activeMasterManager] master.HMaster$1(1546): Failed to become active master java.io.IOException: connection is closed at org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:254) at org.apache.hadoop.hbase.MetaTableAccessor.putToMetaTable(MetaTableAccessor.java:1341) at org.apache.hadoop.hbase.MetaTableAccessor.updateTableState(MetaTableAccessor.java:1691) at org.apache.hadoop.hbase.MetaTableAccessor.updateTableState(MetaTableAccessor.java:1110) at org.apache.hadoop.hbase.master.TableStateManager.udpateMetaState(TableStateManager.java:180) at org.apache.hadoop.hbase.master.TableStateManager.setTableState(TableStateManager.java:67) at org.apache.hadoop.hbase.master.HMaster.assignMeta(HMaster.java:892) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:711) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1542) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:42,952 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=88, retries=350, started=29526 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:42,969 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:42,969 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51197: callId: 2336 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:42,969 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1105 ms ago, cancelled=false, msg= 2015-06-17 11:09:42,983 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=229, retries=350, started=61028 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:43,080 INFO [10.0.0.110,51197,1434564568043_splitLogManager__ChoreService_1] hbase.ScheduledChore(179): Chore: SplitLogManager Timeout Monitor was stopped 2015-06-17 11:09:43,131 INFO [10.0.0.110,51197,1434564568043_ChoreService_1] hbase.ScheduledChore(179): Chore: 10.0.0.110,51197,1434564568043-MemstoreFlusherChore was stopped 2015-06-17 11:09:43,138 INFO [master//10.0.0.110:0.logRoller] regionserver.LogRoller(159): LogRoller exiting. 2015-06-17 11:09:43,138 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(146): master//10.0.0.110:0.leaseChecker closing leases 2015-06-17 11:09:43,138 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(149): master//10.0.0.110:0.leaseChecker closed leases 2015-06-17 11:09:43,153 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=89, retries=350, started=29727 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:43,173 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:43,174 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51197] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51197: callId: 2338 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51202 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:43,174 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=14, retries=35, started=1310 ms ago, cancelled=false, msg= 2015-06-17 11:09:43,185 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=230, retries=350, started=61230 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:43,346 DEBUG [M:0;10.0.0.110:51197] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:09:43,346 INFO [M:0;10.0.0.110:51197] wal.FSHLog(977): Closed WAL: FSHLog 10.0.0.110%2C51197%2C1434564568043.default:(num 1434564568125) 2015-06-17 11:09:43,346 DEBUG [M:0;10.0.0.110:51197] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:09:43,346 INFO [M:0;10.0.0.110:51197] regionserver.Leases(146): M:0;10.0.0.110:51197 closing leases 2015-06-17 11:09:43,346 INFO [M:0;10.0.0.110:51197] regionserver.Leases(149): M:0;10.0.0.110:51197 closed leases 2015-06-17 11:09:43,346 INFO [M:0;10.0.0.110:51197] hbase.ChoreService(302): Chore service for: 10.0.0.110,51197,1434564568043 had [[ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region 10.0.0.110,51197,1434564568043 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS]] on shutdown 2015-06-17 11:09:43,346 INFO [M:0;10.0.0.110:51197] regionserver.CompactSplitThread(394): Waiting for Split Thread to finish... 2015-06-17 11:09:43,346 INFO [M:0;10.0.0.110:51197] regionserver.CompactSplitThread(394): Waiting for Merge Thread to finish... 2015-06-17 11:09:43,346 INFO [M:0;10.0.0.110:51197] regionserver.CompactSplitThread(394): Waiting for Large Compaction Thread to finish... 2015-06-17 11:09:43,347 INFO [M:0;10.0.0.110:51197] regionserver.CompactSplitThread(394): Waiting for Small Compaction Thread to finish... 2015-06-17 11:09:43,348 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51197-0x14e02b3a232003b, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/10.0.0.110,51197,1434564568043 2015-06-17 11:09:43,348 INFO [M:0;10.0.0.110:51197] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a232003f 2015-06-17 11:09:43,349 DEBUG [M:0;10.0.0.110:51197] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:09:43,349 DEBUG [M:0;10.0.0.110:51197] master.HMaster(1074): Stopping service threads 2015-06-17 11:09:43,351 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:09:43,351 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51197-0x14e02b3a232003b, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:09:43,351 INFO [M:0;10.0.0.110:51197] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a232003d 2015-06-17 11:09:43,351 DEBUG [main-EventThread] zookeeper.ZKUtil(488): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:09:43,351 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(488): master:51197-0x14e02b3a232003b, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:09:43,351 DEBUG [M:0;10.0.0.110:51197] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:09:43,352 INFO [M:0;10.0.0.110:51197] hbase.ChoreService(302): Chore service for: 10.0.0.110,51197,1434564568043_splitLogManager_ had [] on shutdown 2015-06-17 11:09:43,352 INFO [M:0;10.0.0.110:51197] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2015-06-17 11:09:43,352 INFO [M:0;10.0.0.110:51197] ipc.RpcServer(2219): Stopping server on 51197 2015-06-17 11:09:43,352 INFO [RpcServer.listener,port=51197] ipc.RpcServer$Listener(776): RpcServer.listener,port=51197: stopping 2015-06-17 11:09:43,352 INFO [RpcServer.responder] ipc.RpcServer$Responder(1018): RpcServer.responder: stopped 2015-06-17 11:09:43,352 INFO [RpcServer.responder] ipc.RpcServer$Responder(921): RpcServer.responder: stopping 2015-06-17 11:09:43,354 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=90, retries=350, started=29928 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:43,354 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,51197,1434564568043 2015-06-17 11:09:43,354 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51197-0x14e02b3a232003b, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,51197,1434564568043 2015-06-17 11:09:43,354 INFO [MasterRestartThread-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [10.0.0.110,51197,1434564568043] 2015-06-17 11:09:43,355 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51197-0x14e02b3a232003b, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:09:43,355 INFO [M:0;10.0.0.110:51197] regionserver.HRegionServer(1093): stopping server 10.0.0.110,51197,1434564568043; zookeeper connection closed. 2015-06-17 11:09:43,355 INFO [M:0;10.0.0.110:51197] regionserver.HRegionServer(1096): M:0;10.0.0.110:51197 exiting 2015-06-17 11:09:43,355 INFO [MasterRestartThread] procedure.TestMasterRestartWithProcedures$5(200): executing master restart 2015-06-17 11:09:43,356 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:43,356 INFO [main-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(121): /hbase/rs/10.0.0.110,51197,1434564568043 znode expired, triggering replicatorRemoved event 2015-06-17 11:09:43,356 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:09:43,356 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:43,371 INFO [MasterRestartThread] client.ConnectionUtils(105): master//10.0.0.110:0 server-side HConnection retries=350 2015-06-17 11:09:43,371 INFO [MasterRestartThread] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2015-06-17 11:09:43,373 INFO [MasterRestartThread] ipc.RpcServer$Listener(598): master//10.0.0.110:0: started 10 reader(s). 2015-06-17 11:09:43,375 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:09:43,376 INFO [MasterRestartThread] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:09:43,376 DEBUG [Thread-199] zookeeper.ZKUtil(745): hconnection-0x51e5f22a-0x14e02b3a2320008, quorum=localhost:51848, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2015-06-17 11:09:43,376 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=15, retries=35, started=1512 ms ago, cancelled=false, msg= 2015-06-17 11:09:43,377 INFO [MasterRestartThread] zookeeper.RecoverableZooKeeper(121): Process identifier=master:51210 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:09:43,379 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:512100x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:09:43,380 DEBUG [MasterRestartThread] zookeeper.RecoverableZooKeeper(595): Node /hbase already exists 2015-06-17 11:09:43,380 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(578): master:51210-0x14e02b3a2320040 connected 2015-06-17 11:09:43,382 DEBUG [MasterRestartThread] zookeeper.ZKUtil(488): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:09:43,382 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2015-06-17 11:09:43,383 INFO [RpcServer.responder] ipc.RpcServer$Responder(917): RpcServer.responder: starting 2015-06-17 11:09:43,383 INFO [RpcServer.listener,port=51210] ipc.RpcServer$Listener(729): RpcServer.listener,port=51210: starting 2015-06-17 11:09:43,383 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=0 queue=0 2015-06-17 11:09:43,383 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=1 queue=0 2015-06-17 11:09:43,383 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=2 queue=0 2015-06-17 11:09:43,383 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=3 queue=0 2015-06-17 11:09:43,383 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): B.default Start Handler index=4 queue=0 2015-06-17 11:09:43,383 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=0 queue=0 2015-06-17 11:09:43,384 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=1 queue=1 2015-06-17 11:09:43,384 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=2 queue=0 2015-06-17 11:09:43,384 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=3 queue=1 2015-06-17 11:09:43,384 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=4 queue=0 2015-06-17 11:09:43,384 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=5 queue=1 2015-06-17 11:09:43,384 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=6 queue=0 2015-06-17 11:09:43,384 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=7 queue=1 2015-06-17 11:09:43,384 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=8 queue=0 2015-06-17 11:09:43,384 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=9 queue=1 2015-06-17 11:09:43,385 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=10 queue=0 2015-06-17 11:09:43,385 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=11 queue=1 2015-06-17 11:09:43,385 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=12 queue=0 2015-06-17 11:09:43,385 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=13 queue=1 2015-06-17 11:09:43,385 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=14 queue=0 2015-06-17 11:09:43,385 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=15 queue=1 2015-06-17 11:09:43,385 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=16 queue=0 2015-06-17 11:09:43,385 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=17 queue=1 2015-06-17 11:09:43,385 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=18 queue=0 2015-06-17 11:09:43,386 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Priority Start Handler index=19 queue=1 2015-06-17 11:09:43,386 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=0 queue=0 2015-06-17 11:09:43,386 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=1 queue=0 2015-06-17 11:09:43,386 DEBUG [MasterRestartThread] ipc.RpcExecutor(114): Replication Start Handler index=2 queue=0 2015-06-17 11:09:43,386 INFO [MasterRestartThread] master.HMaster(349): hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034, hbase.cluster.distributed=false 2015-06-17 11:09:43,386 INFO [MasterRestartThread] master.HMaster(1512): Adding backup master ZNode /hbase/backup-masters/10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,387 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=231, retries=350, started=61432 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:43,387 DEBUG [MasterRestartThread] zookeeper.ZKUtil(486): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,388 INFO [M:0;10.0.0.110:51210] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x248ef7a3 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:09:43,388 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:09:43,388 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2015-06-17 11:09:43,389 DEBUG [10.0.0.110:51210.activeMasterManager] zookeeper.ZKUtil(486): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:09:43,389 DEBUG [M:0;10.0.0.110:51210-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x248ef7a30x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:09:43,390 INFO [10.0.0.110:51210.activeMasterManager] master.ActiveMasterManager(170): Deleting ZNode for /hbase/backup-masters/10.0.0.110,51210,1434564583374 from backup master directory 2015-06-17 11:09:43,390 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(486): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-06-17 11:09:43,390 DEBUG [MasterRestartThread-EventThread] master.ActiveMasterManager(126): A master is now available 2015-06-17 11:09:43,390 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,390 WARN [10.0.0.110:51210.activeMasterManager] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:09:43,390 INFO [10.0.0.110:51210.activeMasterManager] master.ActiveMasterManager(179): Registered Active Master=10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,390 DEBUG [M:0;10.0.0.110:51210] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bbf3597, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:09:43,390 DEBUG [M:0;10.0.0.110:51210] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:09:43,391 DEBUG [M:0;10.0.0.110:51210] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:09:43,391 DEBUG [M:0;10.0.0.110:51210-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x248ef7a3-0x14e02b3a2320041 connected 2015-06-17 11:09:43,392 INFO [M:0;10.0.0.110:51210] regionserver.HRegionServer(774): ClusterId : b7c4b21b-aab2-44b4-8562-2b61af22ad6c 2015-06-17 11:09:43,392 DEBUG [M:0;10.0.0.110:51210] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2015-06-17 11:09:43,392 DEBUG [M:0;10.0.0.110:51210] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:09:43,393 DEBUG [M:0;10.0.0.110:51210] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2015-06-17 11:09:43,393 DEBUG [M:0;10.0.0.110:51210] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2015-06-17 11:09:43,393 DEBUG [M:0;10.0.0.110:51210] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:09:43,394 DEBUG [M:0;10.0.0.110:51210] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2015-06-17 11:09:43,394 INFO [M:0;10.0.0.110:51210] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2015-06-17 11:09:43,395 INFO [M:0;10.0.0.110:51210] compactions.PressureAwareCompactionThroughputController(175): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2015-06-17 11:09:43,395 INFO [M:0;10.0.0.110:51210] regionserver.HRegionServer$CompactionChecker(1504): CompactionChecker runs every 1sec 2015-06-17 11:09:43,395 DEBUG [M:0;10.0.0.110:51210] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b750121, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=10.0.0.110/10.0.0.110:0 2015-06-17 11:09:43,395 DEBUG [M:0;10.0.0.110:51210] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:09:43,395 DEBUG [M:0;10.0.0.110:51210] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:09:43,395 DEBUG [M:0;10.0.0.110:51210] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:M:0;10.0.0.110:51210 2015-06-17 11:09:43,396 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:09:43,396 DEBUG [M:0;10.0.0.110:51210] zookeeper.ZKUtil(486): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,396 INFO [M:0;10.0.0.110:51210] regionserver.RegionServerCoprocessorHost(65): System coprocessor loading is enabled 2015-06-17 11:09:43,396 INFO [M:0;10.0.0.110:51210] regionserver.RegionServerCoprocessorHost(66): Table coprocessor loading is enabled 2015-06-17 11:09:43,396 INFO [M:0;10.0.0.110:51210] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,51210,1434564583374 with port=51210, startcode=1434564583374 2015-06-17 11:09:43,396 DEBUG [M:0;10.0.0.110:51210] regionserver.HRegionServer(2295): Master is not running yet 2015-06-17 11:09:43,397 WARN [M:0;10.0.0.110:51210] regionserver.HRegionServer(900): reportForDuty failed; sleeping and then retrying. 2015-06-17 11:09:43,397 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:43,397 DEBUG [main-EventThread] zookeeper.ZKUtil(486): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,400 DEBUG [10.0.0.110:51210.activeMasterManager] util.FSTableDescriptors(807): Current tableInfoPath = hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2015-06-17 11:09:43,402 DEBUG [10.0.0.110:51210.activeMasterManager] util.FSTableDescriptors(811): TableInfo already exists.. Skipping creation 2015-06-17 11:09:43,404 INFO [10.0.0.110:51210.activeMasterManager] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-06-17 11:09:43,404 DEBUG [10.0.0.110:51210.activeMasterManager] coordination.ZKSplitLogManagerCoordination(885): Distributed log replay=true 2015-06-17 11:09:43,405 INFO [10.0.0.110:51210.activeMasterManager] coordination.ZKSplitLogManagerCoordination(599): Found 0 orphan tasks and 0 rescan nodes 2015-06-17 11:09:43,405 DEBUG [10.0.0.110:51210.activeMasterManager] util.FSTableDescriptors(222): Fetching table descriptors from the filesystem. 2015-06-17 11:09:43,413 INFO [10.0.0.110:51210.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x2ccc925e connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:09:43,415 DEBUG [10.0.0.110:51210.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x2ccc925e0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:09:43,415 DEBUG [10.0.0.110:51210.activeMasterManager] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4be915f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:09:43,415 DEBUG [10.0.0.110:51210.activeMasterManager] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:09:43,415 DEBUG [10.0.0.110:51210.activeMasterManager] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:09:43,416 DEBUG [10.0.0.110:51210.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x2ccc925e-0x14e02b3a2320042 connected 2015-06-17 11:09:43,416 INFO [10.0.0.110:51210.activeMasterManager] balancer.StochasticLoadBalancer(135): loading config 2015-06-17 11:09:43,417 DEBUG [10.0.0.110:51210.activeMasterManager] zookeeper.ZKUtil(488): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2015-06-17 11:09:43,417 DEBUG [10.0.0.110:51210.activeMasterManager] zookeeper.ZKUtil(486): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:43,417 DEBUG [10.0.0.110:51210.activeMasterManager] zookeeper.ZKUtil(486): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,418 DEBUG [10.0.0.110:51210.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:43,418 DEBUG [10.0.0.110:51210.activeMasterManager] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,418 INFO [10.0.0.110:51210.activeMasterManager] master.HMaster(568): Server active/primary master=10.0.0.110,51210,1434564583374, sessionid=0x14e02b3a2320040, setting cluster-up flag (Was=true) 2015-06-17 11:09:43,419 DEBUG [10.0.0.110:51210.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/online-snapshot/acquired already exists 2015-06-17 11:09:43,420 INFO [10.0.0.110:51210.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2015-06-17 11:09:43,420 DEBUG [10.0.0.110:51210.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,421 DEBUG [10.0.0.110:51210.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/flush-table-proc/acquired already exists 2015-06-17 11:09:43,422 INFO [10.0.0.110:51210.activeMasterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2015-06-17 11:09:43,422 DEBUG [10.0.0.110:51210.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(238): Starting the controller for procedure member:10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,422 INFO [10.0.0.110:51210.activeMasterManager] master.MasterCoprocessorHost(86): System coprocessor loading is enabled 2015-06-17 11:09:43,422 DEBUG [10.0.0.110:51210.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_OPEN_REGION-10.0.0.110:51210, corePoolSize=5, maxPoolSize=5 2015-06-17 11:09:43,422 DEBUG [10.0.0.110:51210.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_CLOSE_REGION-10.0.0.110:51210, corePoolSize=5, maxPoolSize=5 2015-06-17 11:09:43,422 DEBUG [10.0.0.110:51210.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_SERVER_OPERATIONS-10.0.0.110:51210, corePoolSize=5, maxPoolSize=5 2015-06-17 11:09:43,423 DEBUG [10.0.0.110:51210.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_META_SERVER_OPERATIONS-10.0.0.110:51210, corePoolSize=5, maxPoolSize=5 2015-06-17 11:09:43,423 DEBUG [10.0.0.110:51210.activeMasterManager] executor.ExecutorService(91): Starting executor service name=M_LOG_REPLAY_OPS-10.0.0.110:51210, corePoolSize=10, maxPoolSize=10 2015-06-17 11:09:43,423 DEBUG [10.0.0.110:51210.activeMasterManager] executor.ExecutorService(91): Starting executor service name=MASTER_TABLE_OPERATIONS-10.0.0.110:51210, corePoolSize=1, maxPoolSize=1 2015-06-17 11:09:43,423 INFO [10.0.0.110:51210.activeMasterManager] procedure2.ProcedureExecutor(446): Starting procedure executor threads=9 2015-06-17 11:09:43,423 INFO [10.0.0.110:51210.activeMasterManager] wal.WALProcedureStore(214): Starting WAL Procedure Store lease recovery 2015-06-17 11:09:43,424 INFO [10.0.0.110:51210.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log 2015-06-17 11:09:43,424 INFO [10.0.0.110:51210.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log after 0ms 2015-06-17 11:09:43,424 DEBUG [10.0.0.110:51210.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000001.log; isDirectory=false; length=14915; replication=1; blocksize=134217728; modification_time=1434564413352; access_time=1434564405122; owner=ndimiduk; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:09:43,426 INFO [10.0.0.110:51210.activeMasterManager] util.FSHDFSUtils(173): Recover lease on dfs file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000012.log 2015-06-17 11:09:43,426 INFO [10.0.0.110:51210.activeMasterManager] util.FSHDFSUtils(262): Recovered lease, attempt=0 on file=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000012.log after 0ms 2015-06-17 11:09:43,427 DEBUG [10.0.0.110:51210.activeMasterManager] wal.WALProcedureStore(765): Opening state-log: FileStatus{path=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000012.log; isDirectory=false; length=60; replication=1; blocksize=134217728; modification_time=1434564582938; access_time=1434564568107; owner=ndimiduk.hfs.11; group=supergroup; permission=rw-r--r--; isSymlink=false} 2015-06-17 11:09:43,431 DEBUG [10.0.0.110:51210.activeMasterManager] wal.WALProcedureStore(610): Roll new state log: 13 2015-06-17 11:09:43,432 INFO [10.0.0.110:51210.activeMasterManager] wal.WALProcedureStore(237): Lease acquired for flushLogId: 13 2015-06-17 11:09:43,433 INFO [10.0.0.110:51210.activeMasterManager] wal.ProcedureWALFormatReader(150): No active entry found in state log hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000012.log. removing it 2015-06-17 11:09:43,434 DEBUG [10.0.0.110:51210.activeMasterManager] procedure2.ProcedureExecutor$1(278): load procedures maxProcId=7 2015-06-17 11:09:43,435 DEBUG [10.0.0.110:51210.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=RUNNABLE isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=7 state=RUNNABLE 2015-06-17 11:09:43,435 DEBUG [10.0.0.110:51210.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DeleteTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=6 state=FINISHED 2015-06-17 11:09:43,435 DEBUG [10.0.0.110:51210.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:09:43,435 DEBUG [10.0.0.110:51210.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=5 state=FINISHED 2015-06-17 11:09:43,435 DEBUG [10.0.0.110:51210.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:09:43,435 DEBUG [10.0.0.110:51210.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: EnableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=4 state=FINISHED 2015-06-17 11:09:43,435 DEBUG [10.0.0.110:51210.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:09:43,435 DEBUG [10.0.0.110:51210.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: DisableTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=3 state=FINISHED 2015-06-17 11:09:43,435 DEBUG [10.0.0.110:51210.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:09:43,435 DEBUG [10.0.0.110:51210.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=testMasterProcsWithMasterRestarting) user=ndimiduk (auth:SIMPLE) id=2 state=FINISHED 2015-06-17 11:09:43,435 DEBUG [10.0.0.110:51210.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:09:43,436 DEBUG [10.0.0.110:51210.activeMasterManager] procedure2.ProcedureExecutor(327): Loading procedure state=FINISHED isFailed=false: CreateTableProcedure (table=hbase:namespace) user=ndimiduk (auth:SIMPLE) id=1 state=FINISHED 2015-06-17 11:09:43,436 DEBUG [10.0.0.110:51210.activeMasterManager] procedure2.ProcedureExecutor(340): The procedure is completed state=FINISHED isFailed=false 2015-06-17 11:09:43,436 DEBUG [10.0.0.110:51210.activeMasterManager] wal.WALProcedureStore(652): Remove log: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/MasterProcWALs/state-00000000000000000012.log 2015-06-17 11:09:43,436 INFO [IPC Server handler 4 on 50895] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741897_1073 127.0.0.1:50896 2015-06-17 11:09:43,438 DEBUG [10.0.0.110:51210.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2015-06-17 11:09:43,438 INFO [10.0.0.110:51210.activeMasterManager] zookeeper.RecoverableZooKeeper(121): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:09:43,440 DEBUG [10.0.0.110:51210.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner0x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:09:43,440 DEBUG [10.0.0.110:51210.activeMasterManager] zookeeper.RecoverableZooKeeper(595): Node /hbase/replication/rs already exists 2015-06-17 11:09:43,440 DEBUG [10.0.0.110:51210.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2015-06-17 11:09:43,441 DEBUG [10.0.0.110:51210.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner 2015-06-17 11:09:43,441 DEBUG [10.0.0.110:51210.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2015-06-17 11:09:43,441 DEBUG [10.0.0.110:51210.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(578): replicationLogCleaner-0x14e02b3a2320043 connected 2015-06-17 11:09:43,441 DEBUG [10.0.0.110:51210.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2015-06-17 11:09:43,442 DEBUG [10.0.0.110:51210.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2015-06-17 11:09:43,442 INFO [10.0.0.110:51210.activeMasterManager] master.ServerManager(1011): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 1, timeout of 4500 ms, interval of 1500 ms. 2015-06-17 11:09:43,442 INFO [M:0;10.0.0.110:51210] regionserver.HRegionServer(2276): reportForDuty to master=10.0.0.110,51210,1434564583374 with port=51210, startcode=1434564583374 2015-06-17 11:09:43,442 INFO [M:0;10.0.0.110:51210] master.ServerManager(459): Registering server=10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,442 INFO [M:0;10.0.0.110:51210] regionserver.HRegionServer(1341): Config from master: hbase.rootdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034 2015-06-17 11:09:43,442 INFO [M:0;10.0.0.110:51210] regionserver.HRegionServer(1341): Config from master: fs.defaultFS=hdfs://localhost:50895 2015-06-17 11:09:43,442 INFO [M:0;10.0.0.110:51210] regionserver.HRegionServer(1341): Config from master: hbase.master.info.port=-1 2015-06-17 11:09:43,442 WARN [M:0;10.0.0.110:51210] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-06-17 11:09:43,442 INFO [M:0;10.0.0.110:51210] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:09:43,442 DEBUG [M:0;10.0.0.110:51210] regionserver.HRegionServer(1603): logdir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,445 DEBUG [M:0;10.0.0.110:51210] regionserver.Replication(142): ReplicationStatisticsThread 300 2015-06-17 11:09:43,445 INFO [M:0;10.0.0.110:51210] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:09:43,447 INFO [M:0;10.0.0.110:51210] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.0.0.110%2C51210%2C1434564583374.default, suffix=, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51210,1434564583374, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:09:43,452 INFO [M:0;10.0.0.110:51210] wal.FSHLog(1365): Slow sync cost: 3 ms, current pipeline: [] 2015-06-17 11:09:43,453 INFO [M:0;10.0.0.110:51210] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51210,1434564583374/10.0.0.110%2C51210%2C1434564583374.default.1434564583447 2015-06-17 11:09:43,453 INFO [M:0;10.0.0.110:51210] regionserver.MetricsRegionServerWrapperImpl(113): Computing regionserver metrics every 5000 milliseconds 2015-06-17 11:09:43,454 DEBUG [M:0;10.0.0.110:51210] executor.ExecutorService(91): Starting executor service name=RS_OPEN_REGION-10.0.0.110:51210, corePoolSize=3, maxPoolSize=3 2015-06-17 11:09:43,454 DEBUG [M:0;10.0.0.110:51210] executor.ExecutorService(91): Starting executor service name=RS_OPEN_META-10.0.0.110:51210, corePoolSize=1, maxPoolSize=1 2015-06-17 11:09:43,454 DEBUG [M:0;10.0.0.110:51210] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_REGION-10.0.0.110:51210, corePoolSize=3, maxPoolSize=3 2015-06-17 11:09:43,454 DEBUG [M:0;10.0.0.110:51210] executor.ExecutorService(91): Starting executor service name=RS_CLOSE_META-10.0.0.110:51210, corePoolSize=1, maxPoolSize=1 2015-06-17 11:09:43,454 DEBUG [M:0;10.0.0.110:51210] executor.ExecutorService(91): Starting executor service name=RS_LOG_REPLAY_OPS-10.0.0.110:51210, corePoolSize=2, maxPoolSize=2 2015-06-17 11:09:43,454 DEBUG [M:0;10.0.0.110:51210] executor.ExecutorService(91): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-10.0.0.110:51210, corePoolSize=3, maxPoolSize=3 2015-06-17 11:09:43,456 DEBUG [M:0;10.0.0.110:51210] zookeeper.ZKUtil(486): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:43,456 DEBUG [M:0;10.0.0.110:51210] zookeeper.ZKUtil(486): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,456 INFO [M:0;10.0.0.110:51210] regionserver.ReplicationSourceManager(229): Current list of replicators: [10.0.0.110,50905,1434564404764, 10.0.0.110,51210,1434564583374] other RSs: [10.0.0.110,50905,1434564404764, 10.0.0.110,51210,1434564583374] 2015-06-17 11:09:43,468 INFO [M:0;10.0.0.110:51210] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x4bc412f7 connecting to ZooKeeper ensemble=localhost:51848 2015-06-17 11:09:43,470 DEBUG [M:0;10.0.0.110:51210-EventThread] zookeeper.ZooKeeperWatcher(494): hconnection-0x4bc412f70x0, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-06-17 11:09:43,471 DEBUG [M:0;10.0.0.110:51210] ipc.AbstractRpcClient(106): Codec=org.apache.hadoop.hbase.codec.KeyValueCodecWithTags@218ad373, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-06-17 11:09:43,471 DEBUG [M:0;10.0.0.110:51210] ipc.AsyncRpcClient(157): Starting async Hbase RPC client 2015-06-17 11:09:43,471 DEBUG [M:0;10.0.0.110:51210] ipc.AsyncRpcClient(168): Use global event loop group NioEventLoopGroup 2015-06-17 11:09:43,471 DEBUG [M:0;10.0.0.110:51210-EventThread] zookeeper.ZooKeeperWatcher(578): hconnection-0x4bc412f7-0x14e02b3a2320044 connected 2015-06-17 11:09:43,483 INFO [M:0;10.0.0.110:51210] regionserver.HeapMemoryManager(190): Starting HeapMemoryTuner chore. 2015-06-17 11:09:43,483 INFO [SplitLogWorker-10.0.0.110:51210] regionserver.SplitLogWorker(134): SplitLogWorker 10.0.0.110,51210,1434564583374 starting 2015-06-17 11:09:43,483 INFO [M:0;10.0.0.110:51210] regionserver.HRegionServer(1363): Serving as 10.0.0.110,51210,1434564583374, RpcServer on 10.0.0.110/10.0.0.110:51210, sessionid=0x14e02b3a2320040 2015-06-17 11:09:43,483 DEBUG [M:0;10.0.0.110:51210] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2015-06-17 11:09:43,483 DEBUG [M:0;10.0.0.110:51210] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager 10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,483 DEBUG [M:0;10.0.0.110:51210] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,51210,1434564583374' 2015-06-17 11:09:43,483 DEBUG [M:0;10.0.0.110:51210] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2015-06-17 11:09:43,484 DEBUG [M:0;10.0.0.110:51210] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2015-06-17 11:09:43,484 DEBUG [M:0;10.0.0.110:51210] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2015-06-17 11:09:43,484 DEBUG [M:0;10.0.0.110:51210] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2015-06-17 11:09:43,484 DEBUG [M:0;10.0.0.110:51210] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager 10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,484 DEBUG [M:0;10.0.0.110:51210] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.0.0.110,51210,1434564583374' 2015-06-17 11:09:43,484 DEBUG [M:0;10.0.0.110:51210] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2015-06-17 11:09:43,484 DEBUG [M:0;10.0.0.110:51210] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2015-06-17 11:09:43,485 DEBUG [M:0;10.0.0.110:51210] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2015-06-17 11:09:43,485 INFO [M:0;10.0.0.110:51210] quotas.RegionServerQuotaManager(62): Quota support disabled 2015-06-17 11:09:43,492 INFO [10.0.0.110:51210.activeMasterManager] master.ServerManager(1028): Finished waiting for region servers count to settle; checked in 1, slept for 50 ms, expecting minimum of 1, maximum of 1, master is running 2015-06-17 11:09:43,493 INFO [10.0.0.110:51210.activeMasterManager] master.ServerManager(459): Registering server=10.0.0.110,50905,1434564404764 2015-06-17 11:09:43,493 INFO [10.0.0.110:51210.activeMasterManager] master.HMaster(665): Registered server found up in zk but who has not yet reported in: 10.0.0.110,50905,1434564404764 2015-06-17 11:09:43,495 INFO [10.0.0.110:51210.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50905,1434564404764 belongs to an existing region server 2015-06-17 11:09:43,498 INFO [10.0.0.110:51210.activeMasterManager] master.MasterFileSystem(265): Log folder hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51210,1434564583374 belongs to an existing region server 2015-06-17 11:09:43,500 DEBUG [10.0.0.110:51210.activeMasterManager] coordination.ZKSplitLogManagerCoordination(650): Marked 1588230740 recovering from 10.0.0.110,51168,1434564553036: /hbase/recovering-regions/1588230740/10.0.0.110,51168,1434564553036 2015-06-17 11:09:43,500 INFO [10.0.0.110:51210.activeMasterManager] master.HMaster(884): Re-assigning hbase:meta with replicaId, 0 it was on 10.0.0.110,51168,1434564553036 2015-06-17 11:09:43,500 DEBUG [10.0.0.110:51210.activeMasterManager] master.AssignmentManager(1239): No previous transition plan found (or ignoring an existing plan) for hbase:meta,,1.1588230740; generated random plan=hri=hbase:meta,,1.1588230740, src=, dest=10.0.0.110,51210,1434564583374; 2 (online=2) available servers, forceNewPlan=false 2015-06-17 11:09:43,500 INFO [10.0.0.110:51210.activeMasterManager] master.AssignmentManager(1028): Assigning hbase:meta,,1.1588230740 to 10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,501 INFO [10.0.0.110:51210.activeMasterManager] master.RegionStates(1072): Transition {1588230740 state=OFFLINE, ts=1434564583500, server=10.0.0.110,51168,1434564553036} to {1588230740 state=PENDING_OPEN, ts=1434564583501, server=10.0.0.110,51210,1434564583374} 2015-06-17 11:09:43,501 INFO [10.0.0.110:51210.activeMasterManager] zookeeper.MetaTableLocator(440): Setting hbase:meta region location in ZooKeeper as 10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,501 DEBUG [10.0.0.110:51210.activeMasterManager] master.ServerManager(937): New admin connection to 10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,501 INFO [10.0.0.110:51210.activeMasterManager] regionserver.RSRpcServices(1432): Open hbase:meta,,1.1588230740 2015-06-17 11:09:43,502 INFO [RS_OPEN_META-10.0.0.110:51210-0] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-06-17 11:09:43,502 DEBUG [10.0.0.110:51210.activeMasterManager] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564583502,"tag":[],"qualifier":"state","vlen":2}]},"row":"hbase:meta"} 2015-06-17 11:09:43,504 INFO [RS_OPEN_META-10.0.0.110:51210-0] wal.FSHLog(511): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.0.0.110%2C51210%2C1434564583374..meta, suffix=.meta, logDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51210,1434564583374, archiveDir=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:09:43,509 INFO [RS_OPEN_META-10.0.0.110:51210-0] wal.FSHLog(1365): Slow sync cost: 3 ms, current pipeline: [] 2015-06-17 11:09:43,509 INFO [RS_OPEN_META-10.0.0.110:51210-0] wal.FSHLog(840): New WAL /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51210,1434564583374/10.0.0.110%2C51210%2C1434564583374..meta.1434564583504.meta 2015-06-17 11:09:43,509 DEBUG [RS_OPEN_META-10.0.0.110:51210-0] regionserver.HRegion(6070): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2015-06-17 11:09:43,510 DEBUG [RS_OPEN_META-10.0.0.110:51210-0] coprocessor.CoprocessorHost(179): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2015-06-17 11:09:43,510 DEBUG [RS_OPEN_META-10.0.0.110:51210-0] regionserver.HRegion(7284): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2015-06-17 11:09:43,510 INFO [RS_OPEN_META-10.0.0.110:51210-0] regionserver.RegionCoprocessorHost(350): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2015-06-17 11:09:43,510 DEBUG [RS_OPEN_META-10.0.0.110:51210-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table meta 1588230740 2015-06-17 11:09:43,511 DEBUG [RS_OPEN_META-10.0.0.110:51210-0] regionserver.HRegion(722): Instantiated hbase:meta,,1.1588230740 2015-06-17 11:09:43,513 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:09:43,513 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:09:43,520 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/b8513b7ab4244f5bb99ff80882e23b4c, isReference=false, isBulkLoadResult=false, seqid=200000177, majorCompaction=false 2015-06-17 11:09:43,526 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/cb290dc49abe4e33b88400230c798a8f, isReference=false, isBulkLoadResult=false, seqid=160000161, majorCompaction=true 2015-06-17 11:09:43,526 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:09:43,527 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:09:43,535 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/9a75bf859fde4bed8e2d6b731b927c46, isReference=false, isBulkLoadResult=false, seqid=160000161, majorCompaction=true 2015-06-17 11:09:43,540 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/ad4ce718ee024b848754720b7c44b2fa, isReference=false, isBulkLoadResult=false, seqid=200000177, majorCompaction=false 2015-06-17 11:09:43,541 DEBUG [RS_OPEN_META-10.0.0.110:51210-0] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740 2015-06-17 11:09:43,543 DEBUG [RS_OPEN_META-10.0.0.110:51210-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:09:43,546 DEBUG [RS_OPEN_META-10.0.0.110:51210-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/recovered.edits/240000180.seqid to file, newSeqId=240000180, maxSeqId=200000180 2015-06-17 11:09:43,546 INFO [RS_OPEN_META-10.0.0.110:51210-0] regionserver.HRegion(848): Onlined 1588230740; next sequenceid=240000180 2015-06-17 11:09:43,547 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1893): Post open deploy tasks for hbase:meta,,1.1588230740 2015-06-17 11:09:43,548 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/recovering-regions/1588230740 2015-06-17 11:09:43,549 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(3181): Update last flushed sequence id of region 1588230740 for 10.0.0.110,51168,1434564553036 2015-06-17 11:09:43,549 DEBUG [PostOpenDeployTasks:1588230740] master.AssignmentManager(2812): Got transition OPENED for {1588230740 state=PENDING_OPEN, ts=1434564583501, server=10.0.0.110,51210,1434564583374} from 10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,549 INFO [PostOpenDeployTasks:1588230740] master.RegionStates(1072): Transition {1588230740 state=PENDING_OPEN, ts=1434564583501, server=10.0.0.110,51210,1434564583374} to {1588230740 state=OPEN, ts=1434564583549, server=10.0.0.110,51210,1434564583374} 2015-06-17 11:09:43,549 INFO [PostOpenDeployTasks:1588230740] zookeeper.MetaTableLocator(440): Setting hbase:meta region location in ZooKeeper as 10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,549 DEBUG [PostOpenDeployTasks:1588230740] master.RegionStates(417): Onlined 1588230740 on 10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,550 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1920): Finished post open deploy task for hbase:meta,,1.1588230740 2015-06-17 11:09:43,550 DEBUG [RS_OPEN_META-10.0.0.110:51210-0] handler.OpenRegionHandler(125): Opened hbase:meta,,1.1588230740 on 10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,555 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=91, retries=350, started=30129 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:43,577 WARN [Thread-199] client.ConnectionImplementation(1751): Checking master connection com.google.protobuf.ServiceException: java.nio.channels.ClosedChannelException at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:224) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:288) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:52299) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1033) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1744) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1390) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:116) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: java.nio.channels.ClosedChannelException 2015-06-17 11:09:43,578 DEBUG [AsyncRpcChannel-pool2-t12] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service MasterService, sasl=false 2015-06-17 11:09:43,578 DEBUG [RpcServer.listener,port=51210] ipc.RpcServer$Listener(840): RpcServer.listener,port=51210: connection from 10.0.0.110:51223; # active connections: 1 2015-06-17 11:09:43,579 INFO [RpcServer.reader=1,bindAddress=10.0.0.110,port=51210] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51223 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:09:43,580 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51210] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:43,580 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51210] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51210: callId: 2342 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51223 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:43,580 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=16, retries=35, started=1716 ms ago, cancelled=false, msg= 2015-06-17 11:09:43,588 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=232, retries=350, started=61633 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:43,704 DEBUG [AsyncRpcChannel-pool2-t13] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service ClientService, sasl=false 2015-06-17 11:09:43,705 DEBUG [RpcServer.listener,port=51210] ipc.RpcServer$Listener(840): RpcServer.listener,port=51210: connection from 10.0.0.110:51224; # active connections: 2 2015-06-17 11:09:43,705 INFO [RpcServer.reader=2,bindAddress=10.0.0.110,port=51210] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51224 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:09:43,707 INFO [10.0.0.110:51210.activeMasterManager] hbase.MetaTableAccessor(1692): Updated table hbase:meta state to ENABLED in META 2015-06-17 11:09:43,709 DEBUG [10.0.0.110:51210.activeMasterManager] master.MasterFileSystem(339): Renamed region directory: hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51168,1434564553036-splitting 2015-06-17 11:09:43,709 INFO [10.0.0.110:51210.activeMasterManager] master.SplitLogManager(560): dead splitlog workers [10.0.0.110,51168,1434564553036] 2015-06-17 11:09:43,709 INFO [10.0.0.110:51210.activeMasterManager] master.SplitLogManager(192): hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51168,1434564553036-splitting is empty dir, no logs to split 2015-06-17 11:09:43,710 DEBUG [10.0.0.110:51210.activeMasterManager] master.SplitLogManager(255): Scheduling batch of logs to split 2015-06-17 11:09:43,710 INFO [10.0.0.110:51210.activeMasterManager] master.SplitLogManager(257): started splitting 0 logs in [hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51168,1434564553036-splitting] for [10.0.0.110,51168,1434564553036] 2015-06-17 11:09:43,710 DEBUG [10.0.0.110:51210.activeMasterManager] coordination.ZKSplitLogManagerCoordination(324): Processing recovering [1588230740] and servers [10.0.0.110,51168,1434564553036], isMetaRecovery=true 2015-06-17 11:09:43,712 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/recovering-regions/1588230740/10.0.0.110,51168,1434564553036 2015-06-17 11:09:43,713 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/recovering-regions/1588230740 2015-06-17 11:09:43,714 INFO [10.0.0.110:51210.activeMasterManager] master.SplitLogManager(316): finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51168,1434564553036-splitting] in 4ms 2015-06-17 11:09:43,714 INFO [RS_OPEN_REGION-10.0.0.110:51210-0] handler.FinishRegionRecoveringHandler(52): /hbase/recovering-regions/1588230740 deleted; 1588230740 recovered. 2015-06-17 11:09:43,714 DEBUG [10.0.0.110:51210.activeMasterManager] hbase.MetaTableAccessor(1352): Put{"totalColumns":1,"families":{"table":[{"timestamp":1434564583714,"tag":[],"qualifier":"state","vlen":2}]},"row":"hbase:meta"} 2015-06-17 11:09:43,718 INFO [10.0.0.110:51210.activeMasterManager] hbase.MetaTableAccessor(1692): Updated table hbase:meta state to ENABLED in META 2015-06-17 11:09:43,725 INFO [10.0.0.110:51210.activeMasterManager] master.ServerManager(686): AssignmentManager hasn't finished failover cleanup; waiting 2015-06-17 11:09:43,725 INFO [10.0.0.110:51210.activeMasterManager] master.HMaster(911): hbase:meta with replicaId 0 assigned=1, location=10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,741 INFO [10.0.0.110:51210.activeMasterManager] master.RegionStates(1072): Transition {9ccc7b1e8942a5472c9cf2d844c2fff0 state=OPEN, ts=1434564583737, server=10.0.0.110,51168,1434564553036} to {9ccc7b1e8942a5472c9cf2d844c2fff0 state=CLOSED, ts=1434564583741, server=10.0.0.110,51168,1434564553036} 2015-06-17 11:09:43,741 INFO [10.0.0.110:51210.activeMasterManager] master.AssignmentManager(538): Clean cluster startup. Assigning user regions 2015-06-17 11:09:43,741 INFO [10.0.0.110:51210.activeMasterManager] master.AssignmentManager(680): Assigning 1 region(s) to 10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,742 INFO [10.0.0.110:51210.activeMasterManager] master.RegionStates(1072): Transition {9ccc7b1e8942a5472c9cf2d844c2fff0 state=CLOSED, ts=1434564583741, server=10.0.0.110,51168,1434564553036} to {9ccc7b1e8942a5472c9cf2d844c2fff0 state=PENDING_OPEN, ts=1434564583742, server=10.0.0.110,51210,1434564583374} 2015-06-17 11:09:43,742 INFO [10.0.0.110:51210.activeMasterManager] master.RegionStateStore(207): Updating row hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. with state=PENDING_OPEN&sn=10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,743 INFO [10.0.0.110:51210.activeMasterManager] regionserver.RSRpcServices(1432): Open hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:09:43,744 DEBUG [10.0.0.110:51210.activeMasterManager] master.AssignmentManager(851): Bulk assigning done for 10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,744 DEBUG [RS_OPEN_REGION-10.0.0.110:51210-1] regionserver.HRegion(6070): Opening region: {ENCODED => 9ccc7b1e8942a5472c9cf2d844c2fff0, NAME => 'hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.', STARTKEY => '', ENDKEY => ''} 2015-06-17 11:09:43,745 DEBUG [RS_OPEN_REGION-10.0.0.110:51210-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table namespace 9ccc7b1e8942a5472c9cf2d844c2fff0 2015-06-17 11:09:43,745 DEBUG [RS_OPEN_REGION-10.0.0.110:51210-1] regionserver.HRegion(722): Instantiated hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:09:43,747 INFO [StoreOpener-9ccc7b1e8942a5472c9cf2d844c2fff0-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=4, currentSize=1077592, freeSize=1042884712, maxSize=1043962304, heapSize=1077592, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-06-17 11:09:43,747 INFO [StoreOpener-9ccc7b1e8942a5472c9cf2d844c2fff0-1] compactions.CompactionConfiguration(99): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-06-17 11:09:43,751 INFO [10.0.0.110:51210.activeMasterManager] master.AssignmentManager(420): Joined the cluster in 25ms, failover=false 2015-06-17 11:09:43,755 DEBUG [10.0.0.110:51210.activeMasterManager] zookeeper.ZKUtil(486): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2015-06-17 11:09:43,755 DEBUG [StoreOpener-9ccc7b1e8942a5472c9cf2d844c2fff0-1] regionserver.HStore(540): loaded hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/info/b88c94ff75ab4fafba9b040819600c6e, isReference=false, isBulkLoadResult=false, seqid=8, majorCompaction=false 2015-06-17 11:09:43,756 DEBUG [10.0.0.110:51210.activeMasterManager] hbase.ZKNamespaceManager(196): Updating namespace cache from node default with data: \x0A\x07default 2015-06-17 11:09:43,756 DEBUG [10.0.0.110:51210.activeMasterManager] hbase.ZKNamespaceManager(196): Updating namespace cache from node hbase with data: \x0A\x05hbase 2015-06-17 11:09:43,756 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=92, retries=350, started=30330 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:43,757 DEBUG [RS_OPEN_REGION-10.0.0.110:51210-1] regionserver.HRegion(3796): Found 0 recovered edits file(s) under hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0 2015-06-17 11:09:43,758 DEBUG [RS_OPEN_REGION-10.0.0.110:51210-1] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-06-17 11:09:43,763 DEBUG [RS_OPEN_REGION-10.0.0.110:51210-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/recovered.edits/27.seqid to file, newSeqId=27, maxSeqId=26 2015-06-17 11:09:43,763 INFO [RS_OPEN_REGION-10.0.0.110:51210-1] regionserver.HRegion(848): Onlined 9ccc7b1e8942a5472c9cf2d844c2fff0; next sequenceid=27 2015-06-17 11:09:43,764 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] regionserver.HRegionServer(1893): Post open deploy tasks for hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:09:43,765 DEBUG [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.AssignmentManager(2812): Got transition OPENED for {9ccc7b1e8942a5472c9cf2d844c2fff0 state=PENDING_OPEN, ts=1434564583742, server=10.0.0.110,51210,1434564583374} from 10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,765 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStates(1072): Transition {9ccc7b1e8942a5472c9cf2d844c2fff0 state=PENDING_OPEN, ts=1434564583742, server=10.0.0.110,51210,1434564583374} to {9ccc7b1e8942a5472c9cf2d844c2fff0 state=OPEN, ts=1434564583765, server=10.0.0.110,51210,1434564583374} 2015-06-17 11:09:43,765 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStateStore(207): Updating row hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. with state=OPEN&openSeqNum=27&server=10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,766 DEBUG [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStates(417): Onlined 9ccc7b1e8942a5472c9cf2d844c2fff0 on 10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,766 INFO [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] master.RegionStates(429): Offlined 9ccc7b1e8942a5472c9cf2d844c2fff0 from 10.0.0.110,51168,1434564553036 2015-06-17 11:09:43,767 DEBUG [PostOpenDeployTasks:9ccc7b1e8942a5472c9cf2d844c2fff0] regionserver.HRegionServer(1920): Finished post open deploy task for hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:09:43,767 DEBUG [RS_OPEN_REGION-10.0.0.110:51210-1] handler.OpenRegionHandler(125): Opened hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. on 10.0.0.110,51210,1434564583374 2015-06-17 11:09:43,782 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51210] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:43,782 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51210] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51210: callId: 2344 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51223 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:43,782 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=17, retries=35, started=1918 ms ago, cancelled=false, msg= 2015-06-17 11:09:43,789 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=233, retries=350, started=61834 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:43,956 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=93, retries=350, started=30530 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:43,983 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51210] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:43,983 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51210] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51210: callId: 2346 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51223 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:43,984 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=18, retries=35, started=2120 ms ago, cancelled=false, msg= 2015-06-17 11:09:43,989 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=234, retries=350, started=62034 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:44,158 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=94, retries=350, started=30731 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:44,186 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51210] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:44,186 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51210] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51210: callId: 2348 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51223 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:44,186 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=19, retries=35, started=2322 ms ago, cancelled=false, msg= 2015-06-17 11:09:44,190 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=235, retries=350, started=62235 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:44,359 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=95, retries=350, started=30933 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:44,388 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51210] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:44,388 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51210] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51210: callId: 2350 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51223 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:44,389 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=20, retries=35, started=2525 ms ago, cancelled=false, msg= 2015-06-17 11:09:44,390 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=236, retries=350, started=62435 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:44,560 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=96, retries=350, started=31134 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:44,590 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=237, retries=350, started=62635 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:44,591 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51210] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:44,591 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51210] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51210: callId: 2352 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51223 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:44,591 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=21, retries=35, started=2727 ms ago, cancelled=false, msg= 2015-06-17 11:09:44,621 DEBUG [AsyncRpcChannel-pool2-t15] ipc.AsyncRpcChannel(487): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2015-06-17 11:09:44,621 DEBUG [RpcServer.listener,port=51210] ipc.RpcServer$Listener(840): RpcServer.listener,port=51210: connection from 10.0.0.110:51227; # active connections: 3 2015-06-17 11:09:44,622 INFO [RpcServer.reader=3,bindAddress=10.0.0.110,port=51210] ipc.RpcServer$Connection(1680): Connection from 10.0.0.110 port: 51227 with version info: version: "2.0.0-SNAPSHOT" url: "git://hw11397.local/Users/ndimiduk/repos/hbase" revision: "623fd63827b2953c150597f24c7205737119bebe" user: "ndimiduk" date: "Wed Jun 17 11:06:16 PDT 2015" src_checksum: "bd81226b9e71f171fefbbe51d883e077" 2015-06-17 11:09:44,762 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=97, retries=350, started=31336 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:44,790 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=238, retries=350, started=62835 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:44,793 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51210] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:44,793 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51210] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51210: callId: 2354 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51223 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:44,793 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=22, retries=35, started=2929 ms ago, cancelled=false, msg= 2015-06-17 11:09:44,963 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=98, retries=350, started=31537 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:44,990 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=239, retries=350, started=63035 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:44,995 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51210] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:44,995 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51210] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51210: callId: 2356 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51223 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:44,996 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=23, retries=35, started=3132 ms ago, cancelled=false, msg= 2015-06-17 11:09:45,076 DEBUG [10.0.0.110,50905,1434564404764_ChoreService_1] compactions.PressureAwareCompactionThroughputController(148): compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-06-17 11:09:45,163 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=99, retries=350, started=31737 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:45,191 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=240, retries=350, started=63236 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:45,198 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51210] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:45,198 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51210] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51210: callId: 2358 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51223 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:45,198 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=24, retries=35, started=3334 ms ago, cancelled=false, msg= 2015-06-17 11:09:45,365 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=100, retries=350, started=31939 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:45,393 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=241, retries=350, started=63438 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:45,400 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51210] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:45,401 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51210] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51210: callId: 2360 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51223 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:45,401 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=25, retries=35, started=3537 ms ago, cancelled=false, msg= 2015-06-17 11:09:45,566 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=101, retries=350, started=32140 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:45,595 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=242, retries=350, started=63640 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:45,603 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51210] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:45,603 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51210] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51210: callId: 2362 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51223 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:45,603 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=26, retries=35, started=3739 ms ago, cancelled=false, msg= 2015-06-17 11:09:45,768 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=102, retries=350, started=32342 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:45,796 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=243, retries=350, started=63841 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:45,804 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51210] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:45,804 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51210] ipc.CallRunner(109): B.defaultRpcServer.handler=3,queue=0,port=51210: callId: 2364 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51223 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:45,805 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=27, retries=35, started=3941 ms ago, cancelled=false, msg= 2015-06-17 11:09:45,969 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=103, retries=350, started=32543 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:45,997 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=244, retries=350, started=64042 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:46,007 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51210] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:46,007 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51210] ipc.CallRunner(109): B.defaultRpcServer.handler=0,queue=0,port=51210: callId: 2366 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51223 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:46,007 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=28, retries=35, started=4143 ms ago, cancelled=false, msg= 2015-06-17 11:09:46,170 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=104, retries=350, started=32744 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:46,199 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=245, retries=350, started=64244 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:46,208 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51210] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:46,209 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51210] ipc.CallRunner(109): B.defaultRpcServer.handler=2,queue=0,port=51210: callId: 2368 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51223 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:46,209 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=29, retries=35, started=4345 ms ago, cancelled=false, msg= 2015-06-17 11:09:46,370 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=105, retries=350, started=32944 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:46,401 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=246, retries=350, started=64446 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:46,412 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51210] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:46,412 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51210] ipc.CallRunner(109): B.defaultRpcServer.handler=4,queue=0,port=51210: callId: 2370 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51223 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:46,412 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=30, retries=35, started=4548 ms ago, cancelled=false, msg= 2015-06-17 11:09:46,572 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=106, retries=350, started=33146 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:46,601 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=247, retries=350, started=64646 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:46,614 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51210] master.MasterRpcServices(987): Checking to see if procedure is done procId=7 2015-06-17 11:09:46,614 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51210] ipc.CallRunner(109): B.defaultRpcServer.handler=1,queue=0,port=51210: callId: 2372 service: MasterService methodName: getProcedureResult size: 29 connection: 10.0.0.110:51223 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2035) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:989) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:49896) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2115) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-06-17 11:09:46,614 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=31, retries=35, started=4750 ms ago, cancelled=false, msg= 2015-06-17 11:09:46,676 WARN [Thread-199] client.HBaseAdmin$ProcedureFuture(4136): failed to get the procedure result procId=7 java.io.InterruptedIOException: Interrupted after 31 tries on 35 at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:165) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) at org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) at org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) at org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2015-06-17 11:09:46,676 INFO [main] hbase.HBaseTestingUtility(1090): Shutting down minicluster 2015-06-17 11:09:46,677 INFO [main] client.ConnectionImplementation(1765): Closing master protocol: MasterService 2015-06-17 11:09:46,677 INFO [main] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320008 2015-06-17 11:09:46,678 DEBUG [main] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:09:46,678 DEBUG [AsyncRpcChannel-pool2-t14] ipc.AsyncRpcChannel$8(574): IPC Client (296483328) connection to /10.0.0.110:51130 from ndimiduk: closed 2015-06-17 11:09:46,678 DEBUG [AsyncRpcChannel-pool2-t3] ipc.AsyncRpcChannel$8(574): IPC Client (-1458831734) connection to /10.0.0.110:51155 from ndimiduk: closed 2015-06-17 11:09:46,678 DEBUG [AsyncRpcChannel-pool2-t4] ipc.AsyncRpcChannel$8(574): IPC Client (1670886626) connection to /10.0.0.110:51047 from ndimiduk: closed 2015-06-17 11:09:46,678 DEBUG [AsyncRpcChannel-pool2-t12] ipc.AsyncRpcChannel$8(574): IPC Client (-686875887) connection to /10.0.0.110:51117 from ndimiduk: closed 2015-06-17 11:09:46,678 DEBUG [AsyncRpcChannel-pool2-t6] ipc.AsyncRpcChannel$8(574): IPC Client (-445393643) connection to /10.0.0.110:50957 from ndimiduk: closed 2015-06-17 11:09:46,678 DEBUG [AsyncRpcChannel-pool2-t7] ipc.AsyncRpcChannel$8(574): IPC Client (-1589588713) connection to /10.0.0.110:51074 from ndimiduk: closed 2015-06-17 11:09:46,678 DEBUG [AsyncRpcChannel-pool2-t9] ipc.AsyncRpcChannel$8(574): IPC Client (1842888695) connection to /10.0.0.110:50975 from ndimiduk: closed 2015-06-17 11:09:46,679 DEBUG [RpcServer.reader=1,bindAddress=10.0.0.110,port=51210] ipc.RpcServer$Listener(872): RpcServer.listener,port=51210: DISCONNECTING client 10.0.0.110:51223 because read count=-1. Number of active connections: 3 2015-06-17 11:09:46,679 DEBUG [AsyncRpcChannel-pool2-t12] ipc.AsyncRpcChannel$8(574): IPC Client (-1064789583) connection to /10.0.0.110:51210 from ndimiduk: closed 2015-06-17 11:09:46,678 DEBUG [AsyncRpcChannel-pool2-t4] ipc.AsyncRpcChannel$8(574): IPC Client (583070302) connection to /10.0.0.110:50903 from ndimiduk: closed 2015-06-17 11:09:46,678 DEBUG [AsyncRpcChannel-pool2-t3] ipc.AsyncRpcChannel$8(574): IPC Client (1097106872) connection to /10.0.0.110:50903 from ndimiduk: closed 2015-06-17 11:09:46,678 DEBUG [main] util.JVMClusterUtil(241): Shutting down HBase Cluster 2015-06-17 11:09:46,678 DEBUG [AsyncRpcChannel-pool2-t16] ipc.AsyncRpcChannel$8(574): IPC Client (1135679198) connection to /10.0.0.110:51015 from ndimiduk: closed 2015-06-17 11:09:46,678 DEBUG [AsyncRpcChannel-pool2-t14] ipc.AsyncRpcChannel$8(574): IPC Client (-80043764) connection to /10.0.0.110:51002 from ndimiduk: closed 2015-06-17 11:09:46,678 DEBUG [AsyncRpcChannel-pool2-t5] ipc.AsyncRpcChannel$8(574): IPC Client (-1243599267) connection to /10.0.0.110:51168 from ndimiduk: closed 2015-06-17 11:09:46,679 INFO [main] regionserver.HRegionServer(1859): STOPPED: Cluster shutdown requested 2015-06-17 11:09:46,679 DEBUG [AsyncRpcChannel-pool2-t9] ipc.AsyncRpcChannel$8(574): IPC Client (-23203093) connection to /10.0.0.110:51197 from ndimiduk: closed 2015-06-17 11:09:46,680 INFO [M:0;10.0.0.110:51210] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2015-06-17 11:09:46,680 INFO [M:0;10.0.0.110:51210] regionserver.HeapMemoryManager(201): Stoping HeapMemoryTuner chore. 2015-06-17 11:09:46,680 INFO [SplitLogWorker-10.0.0.110:51210] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2015-06-17 11:09:46,680 INFO [M:0;10.0.0.110:51210] procedure2.ProcedureExecutor(490): Stopping the procedure executor 2015-06-17 11:09:46,680 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2015-06-17 11:09:46,680 INFO [M:0;10.0.0.110:51210] wal.WALProcedureStore(159): Stopping the WAL Procedure Store 2015-06-17 11:09:46,680 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2015-06-17 11:09:46,680 INFO [SplitLogWorker-10.0.0.110:51210] regionserver.SplitLogWorker(155): SplitLogWorker 10.0.0.110,51210,1434564583374 exiting 2015-06-17 11:09:46,681 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2015-06-17 11:09:46,681 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2015-06-17 11:09:46,681 INFO [main] regionserver.HRegionServer(1859): STOPPED: Shutdown requested 2015-06-17 11:09:46,681 INFO [RS:0;10.0.0.110:50905] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2015-06-17 11:09:46,682 INFO [RS:0;10.0.0.110:50905] regionserver.HRegionServer(975): Stopping infoServer 2015-06-17 11:09:46,682 INFO [SplitLogWorker-10.0.0.110:50905] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2015-06-17 11:09:46,682 INFO [SplitLogWorker-10.0.0.110:50905] regionserver.SplitLogWorker(155): SplitLogWorker 10.0.0.110,50905,1434564404764 exiting 2015-06-17 11:09:46,682 DEBUG [main-EventThread] zookeeper.ZKUtil(488): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2015-06-17 11:09:46,682 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(488): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2015-06-17 11:09:46,685 INFO [RS:0;10.0.0.110:50905] log.Slf4jLog(67): Stopped SelectChannelConnector@0.0.0.0:0 2015-06-17 11:09:46,686 INFO [IPC Server handler 5 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741900_1076{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 0 2015-06-17 11:09:46,687 INFO [M:0;10.0.0.110:51210] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2015-06-17 11:09:46,687 INFO [M:0;10.0.0.110:51210] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2015-06-17 11:09:46,687 INFO [M:0;10.0.0.110:51210] regionserver.HRegionServer(1021): stopping server 10.0.0.110,51210,1434564583374 2015-06-17 11:09:46,687 DEBUG [M:0;10.0.0.110:51210] zookeeper.MetaTableLocator(617): Stopping MetaTableLocator 2015-06-17 11:09:46,687 DEBUG [RS_CLOSE_REGION-10.0.0.110:51210-0] handler.CloseRegionHandler(90): Processing close of hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:09:46,687 DEBUG [RS_CLOSE_REGION-10.0.0.110:51210-0] regionserver.HRegion(1365): Closing hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.: disabling compactions & flushes 2015-06-17 11:09:46,687 DEBUG [RS_CLOSE_REGION-10.0.0.110:51210-0] regionserver.HRegion(1392): Updates disabled for region hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:09:46,688 INFO [M:0;10.0.0.110:51210] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320041 2015-06-17 11:09:46,688 INFO [StoreCloserThread-hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0.-1] regionserver.HStore(870): Closed info 2015-06-17 11:09:46,688 DEBUG [M:0;10.0.0.110:51210] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:09:46,688 INFO [M:0;10.0.0.110:51210] regionserver.CompactSplitThread(394): Waiting for Split Thread to finish... 2015-06-17 11:09:46,689 INFO [M:0;10.0.0.110:51210] regionserver.CompactSplitThread(394): Waiting for Merge Thread to finish... 2015-06-17 11:09:46,689 DEBUG [AsyncRpcChannel-pool2-t14] ipc.AsyncRpcChannel$8(574): IPC Client (-1072988523) connection to /10.0.0.110:51168 from ndimiduk.hfs.12: closed 2015-06-17 11:09:46,689 DEBUG [RpcServer.reader=2,bindAddress=10.0.0.110,port=51210] ipc.RpcServer$Listener(872): RpcServer.listener,port=51210: DISCONNECTING client 10.0.0.110:51224 because read count=-1. Number of active connections: 2 2015-06-17 11:09:46,689 DEBUG [AsyncRpcChannel-pool2-t13] ipc.AsyncRpcChannel$8(574): IPC Client (385211552) connection to /10.0.0.110:51210 from ndimiduk.hfs.12: closed 2015-06-17 11:09:46,689 INFO [M:0;10.0.0.110:51210] regionserver.CompactSplitThread(394): Waiting for Large Compaction Thread to finish... 2015-06-17 11:09:46,689 INFO [M:0;10.0.0.110:51210] regionserver.CompactSplitThread(394): Waiting for Small Compaction Thread to finish... 2015-06-17 11:09:46,689 INFO [M:0;10.0.0.110:51210] regionserver.HRegionServer(1243): Waiting on 2 regions to close 2015-06-17 11:09:46,689 DEBUG [RS_CLOSE_META-10.0.0.110:51210-0] handler.CloseRegionHandler(90): Processing close of hbase:meta,,1.1588230740 2015-06-17 11:09:46,689 DEBUG [M:0;10.0.0.110:51210] regionserver.HRegionServer(1247): {9ccc7b1e8942a5472c9cf2d844c2fff0=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., 1588230740=hbase:meta,,1.1588230740} 2015-06-17 11:09:46,689 DEBUG [RS_CLOSE_META-10.0.0.110:51210-0] regionserver.HRegion(1365): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2015-06-17 11:09:46,689 DEBUG [RS_CLOSE_META-10.0.0.110:51210-0] regionserver.HRegion(1392): Updates disabled for region hbase:meta,,1.1588230740 2015-06-17 11:09:46,690 INFO [RS_CLOSE_META-10.0.0.110:51210-0] regionserver.HRegion(2111): Flushing 2/2 column families, memstore=1.75 KB 2015-06-17 11:09:46,692 DEBUG [RS_CLOSE_REGION-10.0.0.110:51210-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/namespace/9ccc7b1e8942a5472c9cf2d844c2fff0/recovered.edits/29.seqid to file, newSeqId=29, maxSeqId=27 2015-06-17 11:09:46,693 INFO [RS_CLOSE_REGION-10.0.0.110:51210-0] regionserver.HRegion(1486): Closed hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:09:46,693 DEBUG [RS_CLOSE_REGION-10.0.0.110:51210-0] handler.CloseRegionHandler(122): Closed hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0. 2015-06-17 11:09:46,696 INFO [IPC Server handler 6 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741901_1077{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 5462 2015-06-17 11:09:46,774 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=107, retries=350, started=33348 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:46,786 INFO [RS:0;10.0.0.110:50905] regionserver.HeapMemoryManager(201): Stoping HeapMemoryTuner chore. 2015-06-17 11:09:46,786 INFO [RS:0;10.0.0.110:50905] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2015-06-17 11:09:46,787 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2015-06-17 11:09:46,788 INFO [RS:0;10.0.0.110:50905] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2015-06-17 11:09:46,788 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2015-06-17 11:09:46,789 INFO [RS:0;10.0.0.110:50905] regionserver.HRegionServer(1021): stopping server 10.0.0.110,50905,1434564404764 2015-06-17 11:09:46,789 DEBUG [RS:0;10.0.0.110:50905] zookeeper.MetaTableLocator(617): Stopping MetaTableLocator 2015-06-17 11:09:46,789 INFO [RS:0;10.0.0.110:50905] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320002 2015-06-17 11:09:46,791 DEBUG [RS:0;10.0.0.110:50905] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:09:46,791 INFO [RS:0;10.0.0.110:50905] regionserver.HRegionServer(1049): stopping server 10.0.0.110,50905,1434564404764; all regions closed. 2015-06-17 11:09:46,791 DEBUG [RS:0;10.0.0.110:50905] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:46,795 INFO [IPC Server handler 2 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741830_1006{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 5431 2015-06-17 11:09:46,803 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=248, retries=350, started=64848 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:46,975 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=108, retries=350, started=33549 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:47,005 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=249, retries=350, started=65050 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:47,101 INFO [RS_CLOSE_META-10.0.0.110:51210-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=240000191, memsize=1.4 K, hasBloomFilter=false, into tmp file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/eef5e80d8d9e474db552f1becc599f1b 2015-06-17 11:09:47,115 INFO [IPC Server handler 4 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741902_1078{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 0 2015-06-17 11:09:47,116 INFO [RS_CLOSE_META-10.0.0.110:51210-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=240000191, memsize=352, hasBloomFilter=false, into tmp file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/5a8b7c7680b0469f86058b9be5f32a7d 2015-06-17 11:09:47,121 DEBUG [RS_CLOSE_META-10.0.0.110:51210-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/eef5e80d8d9e474db552f1becc599f1b as hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/eef5e80d8d9e474db552f1becc599f1b 2015-06-17 11:09:47,127 INFO [RS_CLOSE_META-10.0.0.110:51210-0] regionserver.HStore(963): Added hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/info/eef5e80d8d9e474db552f1becc599f1b, entries=6, sequenceid=240000191, filesize=5.3 K 2015-06-17 11:09:47,128 DEBUG [RS_CLOSE_META-10.0.0.110:51210-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/.tmp/5a8b7c7680b0469f86058b9be5f32a7d as hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/5a8b7c7680b0469f86058b9be5f32a7d 2015-06-17 11:09:47,133 INFO [RS_CLOSE_META-10.0.0.110:51210-0] regionserver.HStore(963): Added hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/table/5a8b7c7680b0469f86058b9be5f32a7d, entries=2, sequenceid=240000191, filesize=4.6 K 2015-06-17 11:09:47,134 INFO [RS_CLOSE_META-10.0.0.110:51210-0] regionserver.HRegion(2400): Finished memstore flush of ~1.75 KB/1792, currentsize=0 B/0 for region hbase:meta,,1.1588230740 in 445ms, sequenceid=240000191, compaction requested=true 2015-06-17 11:09:47,135 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(870): Closed info 2015-06-17 11:09:47,136 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(870): Closed table 2015-06-17 11:09:47,140 DEBUG [RS_CLOSE_META-10.0.0.110:51210-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:50895/user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/data/hbase/meta/1588230740/recovered.edits/240000194.seqid to file, newSeqId=240000194, maxSeqId=240000180 2015-06-17 11:09:47,141 DEBUG [RS_CLOSE_META-10.0.0.110:51210-0] coprocessor.CoprocessorHost(269): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2015-06-17 11:09:47,141 INFO [RS_CLOSE_META-10.0.0.110:51210-0] regionserver.HRegion(1486): Closed hbase:meta,,1.1588230740 2015-06-17 11:09:47,141 DEBUG [RS_CLOSE_META-10.0.0.110:51210-0] handler.CloseRegionHandler(122): Closed hbase:meta,,1.1588230740 2015-06-17 11:09:47,177 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=109, retries=350, started=33751 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:47,203 DEBUG [RS:0;10.0.0.110:50905] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:09:47,203 INFO [RS:0;10.0.0.110:50905] wal.FSHLog(977): Closed WAL: FSHLog 10.0.0.110%2C50905%2C1434564404764.default:(num 1434564405234) 2015-06-17 11:09:47,203 DEBUG [RS:0;10.0.0.110:50905] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:09:47,203 DEBUG [AsyncRpcChannel-pool2-t13] ipc.AsyncRpcChannel$8(574): IPC Client (-2035209672) connection to /10.0.0.110:51117 from ndimiduk.hfs.0: closed 2015-06-17 11:09:47,203 DEBUG [AsyncRpcChannel-pool2-t15] ipc.AsyncRpcChannel$8(574): IPC Client (1239511570) connection to /10.0.0.110:51002 from ndimiduk.hfs.0: closed 2015-06-17 11:09:47,203 DEBUG [AsyncRpcChannel-pool2-t6] ipc.AsyncRpcChannel$8(574): IPC Client (-2028952688) connection to /10.0.0.110:51047 from ndimiduk.hfs.0: closed 2015-06-17 11:09:47,203 DEBUG [AsyncRpcChannel-pool2-t1] ipc.AsyncRpcChannel$8(574): IPC Client (390823863) connection to /10.0.0.110:50903 from ndimiduk.hfs.0: closed 2015-06-17 11:09:47,203 DEBUG [AsyncRpcChannel-pool2-t10] ipc.AsyncRpcChannel$8(574): IPC Client (1689140394) connection to /10.0.0.110:51074 from ndimiduk.hfs.0: closed 2015-06-17 11:09:47,203 INFO [RS:0;10.0.0.110:50905] regionserver.Leases(146): RS:0;10.0.0.110:50905 closing leases 2015-06-17 11:09:47,204 INFO [RS:0;10.0.0.110:50905] regionserver.Leases(149): RS:0;10.0.0.110:50905 closed leases 2015-06-17 11:09:47,204 DEBUG [AsyncRpcChannel-pool2-t1] ipc.AsyncRpcChannel$8(574): IPC Client (-448357836) connection to /10.0.0.110:51130 from ndimiduk.hfs.0: closed 2015-06-17 11:09:47,204 DEBUG [RpcServer.reader=3,bindAddress=10.0.0.110,port=51210] ipc.RpcServer$Listener(872): RpcServer.listener,port=51210: DISCONNECTING client 10.0.0.110:51227 because read count=-1. Number of active connections: 1 2015-06-17 11:09:47,204 DEBUG [AsyncRpcChannel-pool2-t15] ipc.AsyncRpcChannel$8(574): IPC Client (841895047) connection to /10.0.0.110:51210 from ndimiduk.hfs.0: closed 2015-06-17 11:09:47,204 DEBUG [AsyncRpcChannel-pool2-t3] ipc.AsyncRpcChannel$8(574): IPC Client (236957360) connection to /10.0.0.110:51015 from ndimiduk.hfs.0: closed 2015-06-17 11:09:47,204 DEBUG [AsyncRpcChannel-pool2-t4] ipc.AsyncRpcChannel$8(574): IPC Client (-528191897) connection to /10.0.0.110:51155 from ndimiduk.hfs.0: closed 2015-06-17 11:09:47,203 DEBUG [AsyncRpcChannel-pool2-t8] ipc.AsyncRpcChannel$8(574): IPC Client (-1415086454) connection to /10.0.0.110:50957 from ndimiduk.hfs.0: closed 2015-06-17 11:09:47,203 DEBUG [AsyncRpcChannel-pool2-t12] ipc.AsyncRpcChannel$8(574): IPC Client (-2078682568) connection to /10.0.0.110:50975 from ndimiduk.hfs.0: closed 2015-06-17 11:09:47,203 DEBUG [AsyncRpcChannel-pool2-t11] ipc.AsyncRpcChannel$8(574): IPC Client (-1887253290) connection to /10.0.0.110:51197 from ndimiduk.hfs.0: closed 2015-06-17 11:09:47,204 DEBUG [AsyncRpcChannel-pool2-t8] ipc.AsyncRpcChannel$8(574): IPC Client (1522251511) connection to /10.0.0.110:51168 from ndimiduk.hfs.0: closed 2015-06-17 11:09:47,204 INFO [RS:0;10.0.0.110:50905] hbase.ChoreService(302): Chore service for: 10.0.0.110,50905,1434564404764 had [[ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.0.0.110,50905,1434564404764-MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region 10.0.0.110,50905,1434564404764 Period: 120000 Unit: MILLISECONDS]] on shutdown 2015-06-17 11:09:47,206 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=250, retries=350, started=65251 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:47,292 INFO [M:0;10.0.0.110:51210] regionserver.HRegionServer(1049): stopping server 10.0.0.110,51210,1434564583374; all regions closed. 2015-06-17 11:09:47,292 DEBUG [M:0;10.0.0.110:51210] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51210,1434564583374 2015-06-17 11:09:47,295 INFO [IPC Server handler 8 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741899_1075{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-68dc2820-54ff-46e7-8182-20d6562bff0e:NORMAL|RBW]]} size 83 2015-06-17 11:09:47,298 DEBUG [M:0;10.0.0.110:51210] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:09:47,298 INFO [M:0;10.0.0.110:51210] wal.FSHLog(977): Closed WAL: FSHLog 10.0.0.110%2C51210%2C1434564583374..meta:.meta(num 1434564583504) 2015-06-17 11:09:47,298 DEBUG [M:0;10.0.0.110:51210] wal.FSHLog(1016): Closing WAL writer in /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/WALs/10.0.0.110,51210,1434564583374 2015-06-17 11:09:47,300 INFO [IPC Server handler 2 on 50895] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50896 is added to blk_1073741898_1074{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ddfdf43f-99da-4aaf-8f5c-9e208a9e262d:NORMAL|RBW]]} size 679 2015-06-17 11:09:47,351 INFO [regionserver//10.0.0.110:0.logRoller] regionserver.LogRoller(159): LogRoller exiting. 2015-06-17 11:09:47,351 INFO [regionserver//10.0.0.110:0.leaseChecker] regionserver.Leases(146): regionserver//10.0.0.110:0.leaseChecker closing leases 2015-06-17 11:09:47,351 INFO [RS:0;10.0.0.110:50905] regionserver.CompactSplitThread(394): Waiting for Split Thread to finish... 2015-06-17 11:09:47,351 INFO [regionserver//10.0.0.110:0.leaseChecker] regionserver.Leases(149): regionserver//10.0.0.110:0.leaseChecker closed leases 2015-06-17 11:09:47,351 INFO [RS:0;10.0.0.110:50905] regionserver.CompactSplitThread(394): Waiting for Merge Thread to finish... 2015-06-17 11:09:47,351 INFO [RS:0;10.0.0.110:50905] regionserver.CompactSplitThread(394): Waiting for Large Compaction Thread to finish... 2015-06-17 11:09:47,351 INFO [RS:0;10.0.0.110:50905] regionserver.CompactSplitThread(394): Waiting for Small Compaction Thread to finish... 2015-06-17 11:09:47,353 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:47,353 INFO [RS:0;10.0.0.110:50905] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320006 2015-06-17 11:09:47,354 DEBUG [RS:0;10.0.0.110:50905] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:09:47,354 INFO [RS:0;10.0.0.110:50905] ipc.RpcServer(2219): Stopping server on 50905 2015-06-17 11:09:47,354 INFO [RpcServer.listener,port=50905] ipc.RpcServer$Listener(776): RpcServer.listener,port=50905: stopping 2015-06-17 11:09:47,355 INFO [RpcServer.responder] ipc.RpcServer$Responder(1018): RpcServer.responder: stopped 2015-06-17 11:09:47,355 INFO [RpcServer.responder] ipc.RpcServer$Responder(921): RpcServer.responder: stopping 2015-06-17 11:09:47,357 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:47,357 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,50905,1434564404764 2015-06-17 11:09:47,357 INFO [MasterRestartThread-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [10.0.0.110,50905,1434564404764] 2015-06-17 11:09:47,357 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(494): regionserver:50905-0x14e02b3a2320001, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:09:47,359 INFO [MasterRestartThread-EventThread] master.ServerManager(612): Cluster shutdown set; 10.0.0.110,50905,1434564404764 expired; onlineServers=1 2015-06-17 11:09:47,359 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-06-17 11:09:47,359 INFO [RS:0;10.0.0.110:50905] regionserver.HRegionServer(1093): stopping server 10.0.0.110,50905,1434564404764; zookeeper connection closed. 2015-06-17 11:09:47,359 INFO [RS:0;10.0.0.110:50905] regionserver.HRegionServer(1096): RS:0;10.0.0.110:50905 exiting 2015-06-17 11:09:47,359 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@45eb63c3] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(191): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@45eb63c3 2015-06-17 11:09:47,360 INFO [main] util.JVMClusterUtil(317): Shutdown of 1 master(s) and 1 regionserver(s) complete 2015-06-17 11:09:47,366 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=35, started=489 ms ago, cancelled=false, msg= 2015-06-17 11:09:47,373 INFO [10.0.0.110:51210.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=10, retries=350, started=3617 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51168,1434564553036, seqNum=24 2015-06-17 11:09:47,379 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=110, retries=350, started=33953 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:47,405 INFO [10.0.0.110,51210,1434564583374_splitLogManager__ChoreService_1] hbase.ScheduledChore(179): Chore: SplitLogManager Timeout Monitor was stopped 2015-06-17 11:09:47,408 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=251, retries=350, started=65453 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:47,456 INFO [10.0.0.110,51210,1434564583374_ChoreService_1] hbase.ScheduledChore(179): Chore: 10.0.0.110,51210,1434564583374-MemstoreFlusherChore was stopped 2015-06-17 11:09:47,456 INFO [master//10.0.0.110:0.logRoller] regionserver.LogRoller(159): LogRoller exiting. 2015-06-17 11:09:47,456 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(146): master//10.0.0.110:0.leaseChecker closing leases 2015-06-17 11:09:47,456 INFO [master//10.0.0.110:0.leaseChecker] regionserver.Leases(149): master//10.0.0.110:0.leaseChecker closed leases 2015-06-17 11:09:47,504 INFO [RS_OPEN_META-10.0.0.110:51210-0-MetaLogRoller] regionserver.LogRoller(159): LogRoller exiting. 2015-06-17 11:09:47,568 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=35, started=691 ms ago, cancelled=false, msg= 2015-06-17 11:09:47,575 INFO [10.0.0.110:51210.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=11, retries=350, started=3819 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51168,1434564553036, seqNum=24 2015-06-17 11:09:47,579 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=111, retries=350, started=34153 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:47,609 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=252, retries=350, started=65654 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:47,704 DEBUG [M:0;10.0.0.110:51210] wal.FSHLog(974): Moved 1 WAL file(s) to /user/ndimiduk/test-data/f377e701-7349-47df-ab4c-ff87483ea034/oldWALs 2015-06-17 11:09:47,704 INFO [M:0;10.0.0.110:51210] wal.FSHLog(977): Closed WAL: FSHLog 10.0.0.110%2C51210%2C1434564583374.default:(num 1434564583447) 2015-06-17 11:09:47,704 DEBUG [M:0;10.0.0.110:51210] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:09:47,704 INFO [M:0;10.0.0.110:51210] regionserver.Leases(146): M:0;10.0.0.110:51210 closing leases 2015-06-17 11:09:47,705 INFO [M:0;10.0.0.110:51210] regionserver.Leases(149): M:0;10.0.0.110:51210 closed leases 2015-06-17 11:09:47,705 INFO [M:0;10.0.0.110:51210] hbase.ChoreService(302): Chore service for: 10.0.0.110,51210,1434564583374 had [[ScheduledChore: Name: CatalogJanitor-10.0.0.110:51210 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.0.0.110,51210,1434564583374-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.0.0.110,51210,1434564583374-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region 10.0.0.110,51210,1434564583374 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS]] on shutdown 2015-06-17 11:09:47,706 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/10.0.0.110,51210,1434564583374 2015-06-17 11:09:47,707 INFO [M:0;10.0.0.110:51210] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320044 2015-06-17 11:09:47,707 DEBUG [M:0;10.0.0.110:51210] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:09:47,707 DEBUG [M:0;10.0.0.110:51210] master.HMaster(1074): Stopping service threads 2015-06-17 11:09:47,708 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-06-17 11:09:47,708 INFO [M:0;10.0.0.110:51210] client.ConnectionImplementation(1365): Closing zookeeper sessionid=0x14e02b3a2320042 2015-06-17 11:09:47,709 DEBUG [MasterRestartThread-EventThread] zookeeper.ZKUtil(488): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-06-17 11:09:47,709 DEBUG [M:0;10.0.0.110:51210] ipc.AsyncRpcClient(293): Stopping async HBase RPC client 2015-06-17 11:09:47,709 INFO [M:0;10.0.0.110:51210] hbase.ChoreService(302): Chore service for: 10.0.0.110,51210,1434564583374_splitLogManager_ had [] on shutdown 2015-06-17 11:09:47,709 INFO [M:0;10.0.0.110:51210] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2015-06-17 11:09:47,709 INFO [M:0;10.0.0.110:51210] ipc.RpcServer(2219): Stopping server on 51210 2015-06-17 11:09:47,709 INFO [RpcServer.listener,port=51210] ipc.RpcServer$Listener(776): RpcServer.listener,port=51210: stopping 2015-06-17 11:09:47,709 INFO [RpcServer.responder] ipc.RpcServer$Responder(1018): RpcServer.responder: stopped 2015-06-17 11:09:47,710 INFO [RpcServer.responder] ipc.RpcServer$Responder(921): RpcServer.responder: stopping 2015-06-17 11:09:47,710 DEBUG [MasterRestartThread-EventThread] zookeeper.ZooKeeperWatcher(494): master:51210-0x14e02b3a2320040, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.0.0.110,51210,1434564583374 2015-06-17 11:09:47,710 INFO [MasterRestartThread-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [10.0.0.110,51210,1434564583374] 2015-06-17 11:09:47,711 INFO [M:0;10.0.0.110:51210] regionserver.HRegionServer(1093): stopping server 10.0.0.110,51210,1434564583374; zookeeper connection closed. 2015-06-17 11:09:47,711 INFO [M:0;10.0.0.110:51210] regionserver.HRegionServer(1096): M:0;10.0.0.110:51210 exiting 2015-06-17 11:09:47,713 INFO [main] zookeeper.MiniZooKeeperCluster(316): Shutdown MiniZK cluster with all ZK servers 2015-06-17 11:09:47,714 WARN [main] datanode.DirectoryScanner(375): DirectoryScanner: shutdown has been called 2015-06-17 11:09:47,715 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2015-06-17 11:09:47,770 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=35, started=893 ms ago, cancelled=false, msg= 2015-06-17 11:09:47,776 INFO [10.0.0.110:51210.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=12, retries=350, started=4020 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51168,1434564553036, seqNum=24 2015-06-17 11:09:47,782 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=112, retries=350, started=34356 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:47,809 INFO [10.0.0.110:51130.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=253, retries=350, started=65854 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51074,1434564492618, seqNum=18 2015-06-17 11:09:47,812 DEBUG [10.0.0.110:51197.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner-0x14e02b3a232003e, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2015-06-17 11:09:47,812 DEBUG [10.0.0.110:51155.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner-0x14e02b3a2320034, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2015-06-17 11:09:47,813 DEBUG [10.0.0.110:51155.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(583): replicationLogCleaner-0x14e02b3a2320034, quorum=localhost:51848, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2015-06-17 11:09:47,812 DEBUG [10.0.0.110:51074.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner-0x14e02b3a2320025, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2015-06-17 11:09:47,813 DEBUG [10.0.0.110:51074.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(583): replicationLogCleaner-0x14e02b3a2320025, quorum=localhost:51848, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2015-06-17 11:09:47,812 DEBUG [10.0.0.110:51047.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner-0x14e02b3a2320020, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2015-06-17 11:09:47,814 DEBUG [10.0.0.110:51047.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(583): replicationLogCleaner-0x14e02b3a2320020, quorum=localhost:51848, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2015-06-17 11:09:47,812 DEBUG [10.0.0.110:51002.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner-0x14e02b3a2320016, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2015-06-17 11:09:47,814 DEBUG [10.0.0.110:51002.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(583): replicationLogCleaner-0x14e02b3a2320016, quorum=localhost:51848, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2015-06-17 11:09:47,812 DEBUG [10.0.0.110:51117.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner-0x14e02b3a232002a, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2015-06-17 11:09:47,814 DEBUG [10.0.0.110:51117.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(583): replicationLogCleaner-0x14e02b3a232002a, quorum=localhost:51848, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2015-06-17 11:09:47,812 DEBUG [10.0.0.110:51168.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner-0x14e02b3a2320039, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2015-06-17 11:09:47,814 DEBUG [10.0.0.110:51168.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(583): replicationLogCleaner-0x14e02b3a2320039, quorum=localhost:51848, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2015-06-17 11:09:47,812 DEBUG [10.0.0.110:51015.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner-0x14e02b3a232001b, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2015-06-17 11:09:47,812 DEBUG [10.0.0.110:51210.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner-0x14e02b3a2320043, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2015-06-17 11:09:47,812 DEBUG [10.0.0.110:50957.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner-0x14e02b3a232000c, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2015-06-17 11:09:47,815 DEBUG [10.0.0.110:50957.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(583): replicationLogCleaner-0x14e02b3a232000c, quorum=localhost:51848, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2015-06-17 11:09:47,812 DEBUG [10.0.0.110:50903.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner-0x14e02b3a2320005, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2015-06-17 11:09:47,815 DEBUG [10.0.0.110:50903.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(583): replicationLogCleaner-0x14e02b3a2320005, quorum=localhost:51848, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2015-06-17 11:09:47,812 DEBUG [10.0.0.110:50975.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner-0x14e02b3a2320011, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2015-06-17 11:09:47,815 DEBUG [10.0.0.110:51210.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(583): replicationLogCleaner-0x14e02b3a2320043, quorum=localhost:51848, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2015-06-17 11:09:47,814 DEBUG [10.0.0.110:51015.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(583): replicationLogCleaner-0x14e02b3a232001b, quorum=localhost:51848, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2015-06-17 11:09:47,813 DEBUG [10.0.0.110:51130.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(494): replicationLogCleaner-0x14e02b3a232002f, quorum=localhost:51848, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2015-06-17 11:09:47,812 DEBUG [10.0.0.110:51197.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(583): replicationLogCleaner-0x14e02b3a232003e, quorum=localhost:51848, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2015-06-17 11:09:47,815 DEBUG [10.0.0.110:51130.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(583): replicationLogCleaner-0x14e02b3a232002f, quorum=localhost:51848, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2015-06-17 11:09:47,815 DEBUG [10.0.0.110:50975.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(583): replicationLogCleaner-0x14e02b3a2320011, quorum=localhost:51848, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2015-06-17 11:09:47,818 WARN [DataNode: [[[DISK]file:/Users/ndimiduk/repos/hbase/hbase-server/target/test-data/31bbbe2a-d30b-406a-aae8-a42004014f71/dfscluster_728c1ca4-5c89-445c-9c45-d7f21e719166/dfs/data/data1/, [DISK]file:/Users/ndimiduk/repos/hbase/hbase-server/target/test-data/31bbbe2a-d30b-406a-aae8-a42004014f71/dfscluster_728c1ca4-5c89-445c-9c45-d7f21e719166/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:50895] datanode.BPServiceActor(722): BPOfferService for Block pool BP-746934889-10.0.0.110-1434564402363 (Datanode Uuid 96ae6757-bb52-469b-ab1e-237b0d5d1082) service to localhost/127.0.0.1:50895 interrupted 2015-06-17 11:09:47,818 WARN [DataNode: [[[DISK]file:/Users/ndimiduk/repos/hbase/hbase-server/target/test-data/31bbbe2a-d30b-406a-aae8-a42004014f71/dfscluster_728c1ca4-5c89-445c-9c45-d7f21e719166/dfs/data/data1/, [DISK]file:/Users/ndimiduk/repos/hbase/hbase-server/target/test-data/31bbbe2a-d30b-406a-aae8-a42004014f71/dfscluster_728c1ca4-5c89-445c-9c45-d7f21e719166/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:50895] datanode.BPServiceActor(845): Ending block pool service for: Block pool BP-746934889-10.0.0.110-1434564402363 (Datanode Uuid 96ae6757-bb52-469b-ab1e-237b0d5d1082) service to localhost/127.0.0.1:50895 2015-06-17 11:09:47,824 WARN [org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager$Monitor@44419726] blockmanagement.DecommissionManager$Monitor(78): Monitor interrupted: java.lang.InterruptedException: sleep interrupted 2015-06-17 11:09:47,835 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2015-06-17 11:09:47,957 INFO [main] hbase.HBaseTestingUtility(1103): Minicluster is down 2015-06-17 11:09:47,971 INFO [Thread-199] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=35, started=1094 ms ago, cancelled=false, msg= 2015-06-17 11:09:47,978 INFO [10.0.0.110:51210.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=13, retries=350, started=4222 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51168,1434564553036, seqNum=24 2015-06-17 11:09:47,984 INFO [10.0.0.110:51168.activeMasterManager] client.RpcRetryingCallerImpl(124): Call exception, tries=113, retries=350, started=34558 ms ago, cancelled=false, msg=row 'default' on table 'hbase:namespace' at region=hbase:namespace,,1434564405851.9ccc7b1e8942a5472c9cf2d844c2fff0., hostname=10.0.0.110,51130,1434564521571, seqNum=21 2015-06-17 11:09:48,001 INFO [main] hbase.ResourceChecker(172): after: master.procedure.TestMasterRestartWithProcedures#testMasterProcsWithMasterRestarting Thread=107 (was 4) Potentially hanging thread: 10.0.0.110:50957.activeMasterManager-SendThread(localhost:51848) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) Potentially hanging thread: AsyncRpcChannel-pool2-t4 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: 10.0.0.110:51168.activeMasterManager-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: snapshot-hfile-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: snapshot-log-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: Thread-199 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:160) org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3906) org.apache.hadoop.hbase.client.HBaseAdmin.access$400(HBaseAdmin.java:186) org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:4179) org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4131) org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4087) org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:563) org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:106) org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$1.call(TestMasterRestartWithProcedures.java:101) org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.runRollbackableOperation(TestMasterRestartWithProcedures.java:171) org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures.testMasterProcsWithMasterRestarting(TestMasterRestartWithProcedures.java:101) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Potentially hanging thread: Thread-338 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.master.HMaster$InitializationMonitor.run(HMaster.java:205) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t16 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t14 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t1 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: 10.0.0.110:51074.activeMasterManager-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: org.apache.hadoop.hdfs.PeerCache@7228da38 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:244) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: IPC Client (1162658609) connection to localhost/127.0.0.1:50895 from ndimiduk.hfs.11 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:904) org.apache.hadoop.ipc.Client$Connection.run(Client.java:949) Potentially hanging thread: 10.0.0.110:51074.activeMasterManager-SendThread(localhost:51848) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) Potentially hanging thread: LeaseRenewer:ndimiduk.hfs.10@localhost:50895 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t2 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: snapshot-hfile-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: AsyncRpcChannel-pool2-t5 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-timer-pool1-t1 java.lang.Thread.sleep(Native Method) io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:461) io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:360) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: Thread-386 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.master.HMaster$InitializationMonitor.run(HMaster.java:205) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: 10.0.0.110:51130.activeMasterManager-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: LeaseRenewer:ndimiduk.hfs.8@localhost:50895 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: snapshot-log-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: LeaseRenewer:ndimiduk.hfs.12@localhost:50895 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: snapshot-hfile-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: AsyncRpcChannel-pool2-t8 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: IPC Client (1162658609) connection to localhost/127.0.0.1:50895 from ndimiduk java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:904) org.apache.hadoop.ipc.Client$Connection.run(Client.java:949) Potentially hanging thread: 10.0.0.110:51130.activeMasterManager-SendThread(localhost:51848) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) Potentially hanging thread: snapshot-hfile-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: LeaseRenewer:ndimiduk.hfs.0@localhost:50895 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t9 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: snapshot-log-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: snapshot-log-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: Thread-578 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.master.HMaster$InitializationMonitor.run(HMaster.java:205) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: snapshot-log-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: 10.0.0.110:51197.activeMasterManager-SendThread(localhost:51848) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) Potentially hanging thread: 10.0.0.110:50903.activeMasterManager-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: LeaseRenewer:ndimiduk.hfs.9@localhost:50895 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: 10.0.0.110:51210.activeMasterManager-SendThread(localhost:51848) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) Potentially hanging thread: 10.0.0.110:51155.activeMasterManager-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: AsyncRpcChannel-pool2-t10 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: Thread-833 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.master.HMaster$InitializationMonitor.run(HMaster.java:205) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: Thread-712 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.master.HMaster$InitializationMonitor.run(HMaster.java:205) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: snapshot-hfile-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: snapshot-log-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: 10.0.0.110:51155.activeMasterManager-SendThread(localhost:51848) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) Potentially hanging thread: 10.0.0.110:51047.activeMasterManager-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: LeaseRenewer:ndimiduk.hfs.11@localhost:50895 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t6 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: Timer for 'NameNode' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: AsyncRpcChannel-pool2-t11 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: 10.0.0.110:50975.activeMasterManager-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: AsyncRpcChannel-pool2-t15 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: snapshot-hfile-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: snapshot-log-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: snapshot-hfile-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: 10.0.0.110:51015.activeMasterManager-SendThread(localhost:51848) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) Potentially hanging thread: 10.0.0.110:51002.activeMasterManager-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: AsyncRpcChannel-pool2-t13 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: Thread-457 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.master.HMaster$InitializationMonitor.run(HMaster.java:205) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: snapshot-log-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: snapshot-log-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: 10.0.0.110:51117.activeMasterManager-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: Thread-505 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.master.HMaster$InitializationMonitor.run(HMaster.java:205) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: snapshot-hfile-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: SessionTracker java.lang.Object.wait(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146) Potentially hanging thread: snapshot-hfile-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: 10.0.0.110:50975.activeMasterManager-SendThread(localhost:51848) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) Potentially hanging thread: MasterRestartThread java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.util.Threads.sleep(Threads.java:148) org.apache.hadoop.hbase.MiniHBaseCluster.waitForActiveAndReadyMaster(MiniHBaseCluster.java:485) org.apache.hadoop.hbase.master.procedure.TestMasterRestartWithProcedures$5.run(TestMasterRestartWithProcedures.java:202) Potentially hanging thread: IPC Client (1162658609) connection to localhost/127.0.0.1:50895 from ndimiduk.hfs.0 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:904) org.apache.hadoop.ipc.Client$Connection.run(Client.java:949) Potentially hanging thread: IPC Client (1162658609) connection to localhost/127.0.0.1:50895 from ndimiduk.hfs.12 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:904) org.apache.hadoop.ipc.Client$Connection.run(Client.java:949) Potentially hanging thread: Thread-1013 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.master.HMaster$InitializationMonitor.run(HMaster.java:205) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: 10.0.0.110:51168.activeMasterManager java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:160) org.apache.hadoop.hbase.client.HTable.get(HTable.java:721) org.apache.hadoop.hbase.client.HTable.get(HTable.java:687) org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:137) org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:268) org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:98) org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:919) org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:749) org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169) org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1542) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t7 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: IPC Parameter Sending Thread #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: Thread-760 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.master.HMaster$InitializationMonitor.run(HMaster.java:205) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: 10.0.0.110:51002.activeMasterManager-SendThread(localhost:51848) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) Potentially hanging thread: 10.0.0.110:51197.activeMasterManager-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: snapshot-hfile-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: 10.0.0.110:51210.activeMasterManager java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:160) org.apache.hadoop.hbase.client.HTable.get(HTable.java:721) org.apache.hadoop.hbase.client.HTable.get(HTable.java:687) org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:137) org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:268) org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:98) org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:919) org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:749) org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169) org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1542) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: 10.0.0.110:51130.activeMasterManager java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:160) org.apache.hadoop.hbase.client.HTable.get(HTable.java:721) org.apache.hadoop.hbase.client.HTable.get(HTable.java:687) org.apache.hadoop.hbase.master.TableNamespaceManager.get(TableNamespaceManager.java:137) org.apache.hadoop.hbase.master.TableNamespaceManager.isTableAvailableAndInitialized(TableNamespaceManager.java:268) org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:98) org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:919) org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:749) org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169) org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1542) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: snapshot-log-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: Thread-881 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.master.HMaster$InitializationMonitor.run(HMaster.java:205) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: 10.0.0.110:50903.activeMasterManager-SendThread(localhost:51848) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) Potentially hanging thread: IPC Parameter Sending Thread #1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: 10.0.0.110:50957.activeMasterManager-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: snapshot-log-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: snapshot-log-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: snapshot-log-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: AsyncRpcChannel-pool2-t12 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: Thread-626 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.master.HMaster$InitializationMonitor.run(HMaster.java:205) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t3 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: 10.0.0.110:51168.activeMasterManager-SendThread(localhost:51848) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) Potentially hanging thread: 10.0.0.110:51047.activeMasterManager-SendThread(localhost:51848) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) Potentially hanging thread: Thread-965 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.master.HMaster$InitializationMonitor.run(HMaster.java:205) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: 10.0.0.110:51117.activeMasterManager-SendThread(localhost:51848) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) Potentially hanging thread: ReplicationExecutor-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:560) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: snapshot-hfile-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: 10.0.0.110:51210.activeMasterManager-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: snapshot-hfile-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: 10.0.0.110:51015.activeMasterManager-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: snapshot-hfile-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) - Thread LEAK? -, OpenFileDescriptor=340 (was 179) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=336 (was 500), ProcessCount=253 (was 254), AvailableMemoryMB=13 (was 218) 2015-06-17 11:09:48,005 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@264d0d33 2015-06-17 11:09:48,005 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2015-06-17 11:09:48,005 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@264d0d33 2015-06-17 11:09:48,005 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2015-06-17 11:09:48,005 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@264d0d33 2015-06-17 11:09:48,005 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2015-06-17 11:09:48,005 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@264d0d33 2015-06-17 11:09:48,005 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2015-06-17 11:09:48,005 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@264d0d33 2015-06-17 11:09:48,005 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2015-06-17 11:09:48,005 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@264d0d33 2015-06-17 11:09:48,005 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2015-06-17 11:09:48,005 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@264d0d33 2015-06-17 11:09:48,005 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2015-06-17 11:09:48,005 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@264d0d33 2015-06-17 11:09:48,005 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2015-06-17 11:09:48,005 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@264d0d33 2015-06-17 11:09:48,006 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2015-06-17 11:09:48,006 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@264d0d33 2015-06-17 11:09:48,006 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2015-06-17 11:09:48,006 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@264d0d33 2015-06-17 11:09:48,006 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2015-06-17 11:09:48,006 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@264d0d33 2015-06-17 11:09:48,006 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2015-06-17 11:09:48,006 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@264d0d33 2015-06-17 11:09:48,006 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2015-06-17 11:09:48,006 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@264d0d33 2015-06-17 11:09:48,006 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(120): Starting fs shutdown hook thread. 2015-06-17 11:09:48,006 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished.