2020-06-19 23:47:34,217 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 19mins, 46.261sec 2020-06-19 23:47:39,217 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 19mins, 51.261sec 2020-06-19 23:47:44,218 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 19mins, 56.262sec 2020-06-19 23:47:48,240 INFO [PEWorker-15] client.RpcRetryingCallerImpl: Call exception, tries=17, retries=46, started=1200005 ms ago, cancelled=false, msg=Call to hadoop-hbnn0005.com/172.20.101.36:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=116, waitTime=10763, rpcTimeout=10759, details=row 'abcd2' on table 'hbase:namespace' at region=hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd., hostname=hadoop-hbnn0005.com,16020,1592580274989, seqNum=162, see https://s.apache.org/timeout 2020-06-19 23:47:48,241 WARN [PEWorker-15] procedure.CreateNamespaceProcedure: Retriable error trying to create namespace=abcd2 (in state=CREATE_NAMESPACE_INSERT_INTO_NS_TABLE) java.net.SocketTimeoutException: callTimeout=1200000, callDuration=1220061: Call to hadoop-hbnn0005.com/172.20.101.36:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=116, waitTime=10763, rpcTimeout=10759 row 'abcd2' on table 'hbase:namespace' at region=hbase:namespace,,1592548148073.f5c7e71fb5e5cab3b27e52600996f7fd., hostname=hadoop-hbnn0005.com,16020,1592580274989, seqNum=162 at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:159) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:542) at org.apache.hadoop.hbase.master.TableNamespaceManager.insertIntoNSTable(TableNamespaceManager.java:167) at org.apache.hadoop.hbase.master.procedure.CreateNamespaceProcedure.insertIntoNSTable(CreateNamespaceProcedure.java:240) at org.apache.hadoop.hbase.master.procedure.CreateNamespaceProcedure.executeFromState(CreateNamespaceProcedure.java:85) at org.apache.hadoop.hbase.master.procedure.CreateNamespaceProcedure.executeFromState(CreateNamespaceProcedure.java:39) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:189) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:965) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1723) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1462) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1200(ProcedureExecutor.java:78) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2039) Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call to hadoop-hbnn0005.com/172.20.101.36:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=116, waitTime=10763, rpcTimeout=10759 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:205) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.setTimeout(Call.java:96) at org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:199) at org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:682) at org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:757) at org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:485) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=116, waitTime=10763, rpcTimeout=10759 at org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:200) ... 4 more 2020-06-19 23:47:49,218 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 20mins, 1.262sec 2020-06-19 23:47:54,220 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 20mins, 6.263sec 2020-06-19 23:47:59,220 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 20mins, 11.264sec 2020-06-19 23:48:04,220 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 20mins, 16.264sec 2020-06-19 23:48:09,221 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 20mins, 21.265sec 2020-06-19 23:48:14,221 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 20mins, 26.265sec 2020-06-19 23:48:19,221 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 20mins, 31.265sec 2020-06-19 23:48:24,222 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 20mins, 36.266sec 2020-06-19 23:48:29,222 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 20mins, 41.266sec 2020-06-19 23:48:34,223 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 20mins, 46.267sec 2020-06-19 23:48:39,223 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 20mins, 51.267sec 2020-06-19 23:48:44,223 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 20mins, 56.267sec 2020-06-19 23:48:49,223 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 21mins, 1.267sec 2020-06-19 23:48:54,223 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 21mins, 6.267sec 2020-06-19 23:48:59,223 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 21mins, 11.267sec 2020-06-19 23:49:04,223 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 21mins, 16.267sec 2020-06-19 23:49:09,224 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 21mins, 21.268sec 2020-06-19 23:49:14,224 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 21mins, 26.268sec 2020-06-19 23:49:19,225 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 21mins, 31.269sec 2020-06-19 23:49:24,225 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 21mins, 36.269sec 2020-06-19 23:49:29,225 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 21mins, 41.269sec 2020-06-19 23:49:34,225 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 21mins, 46.269sec 2020-06-19 23:49:39,225 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 21mins, 51.269sec 2020-06-19 23:49:44,225 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 21mins, 56.269sec 2020-06-19 23:49:49,225 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 22mins, 1.269sec 2020-06-19 23:49:54,225 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 22mins, 6.269sec 2020-06-19 23:49:59,225 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 22mins, 11.269sec 2020-06-19 23:50:04,225 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 22mins, 16.269sec 2020-06-19 23:50:09,226 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 22mins, 21.269sec 2020-06-19 23:50:14,226 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 22mins, 26.27sec 2020-06-19 23:50:19,226 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 22mins, 31.27sec 2020-06-19 23:50:24,227 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 22mins, 36.27sec 2020-06-19 23:50:29,227 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 22mins, 41.271sec 2020-06-19 23:50:34,228 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 22mins, 46.271sec 2020-06-19 23:50:39,228 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 22mins, 51.272sec 2020-06-19 23:50:44,228 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 22mins, 56.272sec 2020-06-19 23:50:49,228 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 23mins, 1.272sec 2020-06-19 23:50:54,228 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 23mins, 6.272sec 2020-06-19 23:50:59,228 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 23mins, 11.272sec 2020-06-19 23:51:03,029 INFO [ReadOnlyZKClient-hadoop-hbnn0001.com:2181,hadoop-hbnn0002.com:2181,hadoop-hbnn0003.com:2181@0x37631627] zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-hbnn0001.com:2181,hadoop-hbnn0002.com:2181,hadoop-hbnn0003.com:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$54/2130429099@703c2504 2020-06-19 23:51:03,031 INFO [ReadOnlyZKClient-hadoop-hbnn0001.com:2181,hadoop-hbnn0002.com:2181,hadoop-hbnn0003.com:2181@0x37631627-SendThread(hadoop-hbnn0001.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server hadoop-hbnn0001.com/172.20.101.35:2181. Will not attempt to authenticate using SASL (unknown error) 2020-06-19 23:51:03,033 INFO [ReadOnlyZKClient-hadoop-hbnn0001.com:2181,hadoop-hbnn0002.com:2181,hadoop-hbnn0003.com:2181@0x37631627-SendThread(hadoop-hbnn0001.com:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /172.20.101.35:32960, server: hadoop-hbnn0001.com/172.20.101.35:2181 2020-06-19 23:51:03,036 INFO [ReadOnlyZKClient-hadoop-hbnn0001.com:2181,hadoop-hbnn0002.com:2181,hadoop-hbnn0003.com:2181@0x37631627-SendThread(hadoop-hbnn0001.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server hadoop-hbnn0001.com/172.20.101.35:2181, sessionid = 0x100035d0717007a, negotiated timeout = 40000 2020-06-19 23:51:04,229 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 23mins, 16.273sec 2020-06-19 23:51:09,229 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 23mins, 21.273sec 2020-06-19 23:51:14,229 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 23mins, 26.273sec 2020-06-19 23:51:19,230 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 23mins, 31.274sec 2020-06-19 23:51:24,230 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 23mins, 36.274sec 2020-06-19 23:51:29,231 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 23mins, 41.274sec 2020-06-19 23:51:34,231 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 23mins, 46.275sec 2020-06-19 23:51:39,231 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 23mins, 51.275sec 2020-06-19 23:51:44,231 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 23mins, 56.275sec 2020-06-19 23:51:49,231 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 24mins, 1.275sec 2020-06-19 23:51:54,231 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 24mins, 6.275sec 2020-06-19 23:51:59,232 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 24mins, 11.276sec 2020-06-19 23:52:03,047 INFO [ReadOnlyZKClient-hadoop-hbnn0001.com:2181,hadoop-hbnn0002.com:2181,hadoop-hbnn0003.com:2181@0x37631627] zookeeper.ZooKeeper: Session: 0x100035d0717007a closed 2020-06-19 23:52:03,047 INFO [ReadOnlyZKClient-hadoop-hbnn0001.com:2181,hadoop-hbnn0002.com:2181,hadoop-hbnn0003.com:2181@0x37631627-EventThread] zookeeper.ClientCnxn: EventThread shut down 2020-06-19 23:52:04,232 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 24mins, 16.276sec 2020-06-19 23:52:09,232 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 24mins, 21.276sec 2020-06-19 23:52:14,233 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 24mins, 26.277sec 2020-06-19 23:52:19,233 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 24mins, 31.277sec 2020-06-19 23:52:24,233 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 24mins, 36.277sec 2020-06-19 23:52:29,234 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 24mins, 41.278sec 2020-06-19 23:52:34,234 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 24mins, 46.278sec 2020-06-19 23:52:39,234 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 24mins, 51.278sec 2020-06-19 23:52:44,235 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 24mins, 56.279sec 2020-06-19 23:52:49,235 WARN [ProcExecTimeout] procedure2.ProcedureExecutor: Worker stuck PEWorker-15(pid=171), run time 25mins, 1.279sec