INCORRECT PARAMETER - HELP/USAGE TEXT /hbase$ ~/Applications/hbase/bin/hbase org.apache.hadoop.hbase.tool.Canary -zookeeper -treatFailureAsError -permitZookeeperFailures 2 2018-08-27 15:03:34,642 INFO [main] tool.Canary: Number of execution threads 16 -permitZookeeperFailures options is invalid. Usage: hbase canary [opts] [table1 [table2]...] | [regionserver1 [regionserver2]..] where [opts] are: -help Show this help and exit. -regionserver replace the table argument to regionserver, which means to enable regionserver mode -allRegions Tries all regions on a regionserver, only works in regionserver mode. -zookeeper Tries to grab zookeeper.znode.parent on each zookeeper instance -permittedZookeeperFailures Ignore first N failures when attempting to connect to individual zookeeper nodes in the ensemble -daemon Continuous check at defined intervals. -interval Interval between checks (sec) -e Use table/regionserver as regular expression which means the table/regionserver is regular expression pattern -f stop whole program if first error occurs, default is true -t timeout for a check, default is 600000 (millisecs) -writeTableTimeout write timeout for the writeTable, default is 600000 (millisecs) -readTableTimeouts =,=, ... comma-separated list of read timeouts per table (no spaces), default is 600000 (millisecs) -writeSniffing enable the write sniffing in canary -treatFailureAsError treats read / write failure as error -writeTable The table used for write sniffing. Default is hbase:canary -Dhbase.canary.read.raw.enabled= Use this flag to enable or disable raw scan during read canary test Default is false and raw is not enabled during scan -D= assigning or override the configuration params 5-NODE ZK ENSEMBLE, 2 NODES DOWN, 1 PERMITTED FAILURE, CANARY FAILS :~/work/src/github.com/hbase$ ~/Applications/hbase/bin/hbase org.apache.hadoop.hbase.tool.Canary -zookeeper -treatFailureAsError -permittedZookeeperFailures 1 2018-08-27 15:10:28,769 INFO [main] tool.Canary: Number of execution threads 16 2018-08-27 15:10:28,809 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-08-27 15:10:29,075 INFO [main] zookeeper.ReadOnlyZKClient: Connect 0x78b729e6 to 127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-08-27 15:10:29,093 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185@0x78b729e6] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT 2018-08-27 15:10:29,093 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185@0x78b729e6] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_181 2018-08-27 15:10:29,093 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185@0x78b729e6] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2018-08-27 15:10:29,093 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185@0x78b729e6] zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre 2018-08-27 15:10:29,094 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185@0x78b729e6] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2018-08-27 15:10:29,094 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185@0x78b729e6] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2018-08-27 15:10:29,094 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185@0x78b729e6] zookeeper.ZooKeeper: Client environment:java.compiler= 2018-08-27 15:10:29,094 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185@0x78b729e6] zookeeper.ZooKeeper: Client environment:os.name=Linux 2018-08-27 15:10:29,094 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185@0x78b729e6] zookeeper.ZooKeeper: Client environment:os.arch=amd64 2018-08-27 15:10:29,094 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185@0x78b729e6] zookeeper.ZooKeeper: Client environment:os.version=4.4.0-134-generic 2018-08-27 15:10:29,097 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185@0x78b729e6] zookeeper.ZooKeeper: Initiating client connection, connectString=127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$13/788386517@5022a1e1 2018-08-27 15:10:29,127 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185@0x78b729e6-SendThread(127.0.0.1:2185)] zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2185. Will not attempt to authenticate using SASL (unknown error) 2018-08-27 15:10:29,136 WARN [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185@0x78b729e6-SendThread(127.0.0.1:2185)] zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) 2018-08-27 15:10:29,242 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185@0x78b729e6-SendThread(127.0.0.1:2184)] zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2184. Will not attempt to authenticate using SASL (unknown error) 2018-08-27 15:10:29,243 WARN [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185@0x78b729e6-SendThread(127.0.0.1:2184)] zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) 2018-08-27 15:10:29,245 WARN [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185@0x78b729e6] zookeeper.ReadOnlyZKClient: 0x78b729e6 to 127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185 failed for get of /hbase/hbaseid, code = CONNECTIONLOSS, retries = 1 2018-08-27 15:10:29,344 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185@0x78b729e6-SendThread(127.0.0.1:2183)] zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2183. Will not attempt to authenticate using SASL (unknown error) 2018-08-27 15:10:29,345 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185@0x78b729e6-SendThread(127.0.0.1:2183)] zookeeper.ClientCnxn: Socket connection established to 127.0.0.1/127.0.0.1:2183, initiating session 2018-08-27 15:10:29,365 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185@0x78b729e6-SendThread(127.0.0.1:2183)] zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2183, sessionid = 0x3000eb98adc000c, negotiated timeout = 40000 2018-08-27 15:10:30,259 WARN [main] client.ConnectionImplementation: Retrieve cluster id failed java.util.concurrent.ExecutionException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/hbaseid at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:542) at org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:297) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.client.ConnectionFactory.lambda$createConnection$0(ConnectionFactory.java:229) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:347) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:227) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:127) at org.apache.hadoop.hbase.tool.Canary.run(Canary.java:798) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1609) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/hbaseid at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:174) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) at java.lang.Thread.run(Thread.java:748) 2018-08-27 15:10:30,421 INFO [pool-1-thread-1] zookeeper.ZooKeeper: Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.EmptyWatcher@1ba8e248 2018-08-27 15:10:30,422 INFO [pool-1-thread-2] zookeeper.ZooKeeper: Initiating client connection, connectString=127.0.0.1:2182 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.EmptyWatcher@1ba8e248 2018-08-27 15:10:30,422 INFO [pool-1-thread-3] zookeeper.ZooKeeper: Initiating client connection, connectString=127.0.0.1:2183 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.EmptyWatcher@1ba8e248 2018-08-27 15:10:30,423 INFO [pool-1-thread-1-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2018-08-27 15:10:30,423 INFO [pool-1-thread-1-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session 2018-08-27 15:10:30,424 INFO [pool-1-thread-4] zookeeper.ZooKeeper: Initiating client connection, connectString=127.0.0.1:2184 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.EmptyWatcher@1ba8e248 2018-08-27 15:10:30,424 INFO [pool-1-thread-5] zookeeper.ZooKeeper: Initiating client connection, connectString=127.0.0.1:2185 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.EmptyWatcher@1ba8e248 2018-08-27 15:10:30,425 INFO [pool-1-thread-2-SendThread(127.0.0.1:2182)] zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2182. Will not attempt to authenticate using SASL (unknown error) 2018-08-27 15:10:30,425 INFO [pool-1-thread-3-SendThread(127.0.0.1:2183)] zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2183. Will not attempt to authenticate using SASL (unknown error) 2018-08-27 15:10:30,426 INFO [pool-1-thread-2-SendThread(127.0.0.1:2182)] zookeeper.ClientCnxn: Socket connection established to 127.0.0.1/127.0.0.1:2182, initiating session 2018-08-27 15:10:30,426 INFO [pool-1-thread-3-SendThread(127.0.0.1:2183)] zookeeper.ClientCnxn: Socket connection established to 127.0.0.1/127.0.0.1:2183, initiating session 2018-08-27 15:10:30,427 INFO [pool-1-thread-4-SendThread(127.0.0.1:2184)] zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2184. Will not attempt to authenticate using SASL (unknown error) 2018-08-27 15:10:30,427 WARN [pool-1-thread-4-SendThread(127.0.0.1:2184)] zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) 2018-08-27 15:10:30,428 INFO [pool-1-thread-5-SendThread(127.0.0.1:2185)] zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2185. Will not attempt to authenticate using SASL (unknown error) 2018-08-27 15:10:30,429 WARN [pool-1-thread-5-SendThread(127.0.0.1:2185)] zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) 2018-08-27 15:10:30,460 INFO [pool-1-thread-1-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x1000eb86e96000e, negotiated timeout = 40000 2018-08-27 15:10:30,471 INFO [pool-1-thread-2-SendThread(127.0.0.1:2182)] zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2182, sessionid = 0x2000eb86ebb0012, negotiated timeout = 40000 2018-08-27 15:10:30,471 INFO [pool-1-thread-1] tool.Canary: Read from zNode:/hbase on zookeeper instance:127.0.0.1:2181 in 5ms 2018-08-27 15:10:30,471 INFO [pool-1-thread-3-SendThread(127.0.0.1:2183)] zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2183, sessionid = 0x3000eb98adc000d, negotiated timeout = 40000 2018-08-27 15:10:30,474 INFO [pool-1-thread-3] tool.Canary: Read from zNode:/hbase on zookeeper instance:127.0.0.1:2183 in 1ms 2018-08-27 15:10:30,487 INFO [pool-1-thread-1] zookeeper.ZooKeeper: Session: 0x1000eb86e96000e closed 2018-08-27 15:10:30,487 INFO [pool-1-thread-1-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x1000eb86e96000e 2018-08-27 15:10:30,507 INFO [pool-1-thread-3] zookeeper.ZooKeeper: Session: 0x3000eb98adc000d closed 2018-08-27 15:10:30,507 INFO [pool-1-thread-2] tool.Canary: Read from zNode:/hbase on zookeeper instance:127.0.0.1:2182 in 19ms 2018-08-27 15:10:30,507 INFO [pool-1-thread-3-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x3000eb98adc000d 2018-08-27 15:10:30,522 INFO [pool-1-thread-2] zookeeper.ZooKeeper: Session: 0x2000eb86ebb0012 closed 2018-08-27 15:10:30,522 INFO [pool-1-thread-2-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x2000eb86ebb0012 2018-08-27 15:10:30,528 ERROR [pool-1-thread-4] tool.Canary: Read from zNode:/hbase on zookeeper instance:127.0.0.1:2184 org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase 2018-08-27 15:10:30,530 ERROR [pool-1-thread-5] tool.Canary: Read from zNode:/hbase on zookeeper instance:127.0.0.1:2185 org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase 2018-08-27 15:10:31,528 INFO [pool-1-thread-4-SendThread(127.0.0.1:2184)] zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2184. Will not attempt to authenticate using SASL (unknown error) 2018-08-27 15:10:31,530 INFO [pool-1-thread-5-SendThread(127.0.0.1:2185)] zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2185. Will not attempt to authenticate using SASL (unknown error) 2018-08-27 15:10:31,629 INFO [pool-1-thread-4] zookeeper.ZooKeeper: Session: 0x0 closed 2018-08-27 15:10:31,629 INFO [pool-1-thread-4-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x0 2018-08-27 15:10:31,631 INFO [pool-1-thread-5] zookeeper.ZooKeeper: Session: 0x0 closed 2018-08-27 15:10:31,631 INFO [pool-1-thread-5-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x0 2018-08-27 15:10:32,419 ERROR [main] tool.Canary: Too many failures detected, failing the Canary. 2018-08-27 15:10:32,420 INFO [main] zookeeper.ReadOnlyZKClient: Close zookeeper connection 0x78b729e6 to 127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184,127.0.0.1:2185 3-NODE ZK ENSEMBLE, ALL NODES UP, 1 PERMITTED FAILURE, CANARY SUCCESS :~/work/src/github.com/hbase$ ~/Applications/hbase/bin/hbase org.apache.hadoop.hbase.tool.Canary -zookeeper -treatFailureAsError -permittedZookeeperFailures 1 2018-08-27 15:11:04,578 INFO [main] tool.Canary: Number of execution threads 16 2018-08-27 15:11:04,607 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-08-27 15:11:04,816 INFO [main] zookeeper.ReadOnlyZKClient: Connect 0x78b729e6 to 127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-08-27 15:11:04,828 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT 2018-08-27 15:11:04,828 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_181 2018-08-27 15:11:04,829 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2018-08-27 15:11:04,829 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6] zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre 2018-08-27 15:11:04,829 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2018-08-27 15:11:04,829 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6] zookeeper.ZooKeeper: Client environment:java.compiler= 2018-08-27 15:11:04,829 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6] zookeeper.ZooKeeper: Client environment:os.name=Linux 2018-08-27 15:11:04,829 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6] zookeeper.ZooKeeper: Client environment:os.arch=amd64 2018-08-27 15:11:04,829 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6] zookeeper.ZooKeeper: Client environment:os.version=4.4.0-134-generic 2018-08-27 15:11:04,831 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6] zookeeper.ZooKeeper: Initiating client connection, connectString=127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$13/788386517@5022a1e1 2018-08-27 15:11:04,849 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6-SendThread(127.0.0.1:2182)] zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2182. Will not attempt to authenticate using SASL (unknown error) 2018-08-27 15:11:04,856 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6-SendThread(127.0.0.1:2182)] zookeeper.ClientCnxn: Socket connection established to 127.0.0.1/127.0.0.1:2182, initiating session 2018-08-27 15:11:04,876 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6-SendThread(127.0.0.1:2182)] zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2182, sessionid = 0x2000eb86ebb0013, negotiated timeout = 40000 2018-08-27 15:11:04,893 WARN [main] client.ConnectionImplementation: Retrieve cluster id failed java.util.concurrent.ExecutionException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/hbaseid at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:542) at org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:297) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.client.ConnectionFactory.lambda$createConnection$0(ConnectionFactory.java:229) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:347) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:227) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:127) at org.apache.hadoop.hbase.tool.Canary.run(Canary.java:798) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1609) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/hbaseid at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:174) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) at java.lang.Thread.run(Thread.java:748) 2018-08-27 15:11:05,007 INFO [pool-1-thread-1] zookeeper.ZooKeeper: Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.EmptyWatcher@1e598625 2018-08-27 15:11:05,007 INFO [pool-1-thread-2] zookeeper.ZooKeeper: Initiating client connection, connectString=127.0.0.1:2182 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.EmptyWatcher@1e598625 2018-08-27 15:11:05,007 INFO [pool-1-thread-3] zookeeper.ZooKeeper: Initiating client connection, connectString=127.0.0.1:2183 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.EmptyWatcher@1e598625 2018-08-27 15:11:05,009 INFO [pool-1-thread-1-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2018-08-27 15:11:05,009 INFO [pool-1-thread-1-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session 2018-08-27 15:11:05,010 INFO [pool-1-thread-2-SendThread(127.0.0.1:2182)] zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2182. Will not attempt to authenticate using SASL (unknown error) 2018-08-27 15:11:05,011 INFO [pool-1-thread-2-SendThread(127.0.0.1:2182)] zookeeper.ClientCnxn: Socket connection established to 127.0.0.1/127.0.0.1:2182, initiating session 2018-08-27 15:11:05,011 INFO [pool-1-thread-3-SendThread(127.0.0.1:2183)] zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2183. Will not attempt to authenticate using SASL (unknown error) 2018-08-27 15:11:05,011 INFO [pool-1-thread-3-SendThread(127.0.0.1:2183)] zookeeper.ClientCnxn: Socket connection established to 127.0.0.1/127.0.0.1:2183, initiating session 2018-08-27 15:11:05,019 INFO [pool-1-thread-1-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x1000eb86e96000f, negotiated timeout = 40000 2018-08-27 15:11:05,024 INFO [pool-1-thread-2-SendThread(127.0.0.1:2182)] zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2182, sessionid = 0x2000eb86ebb0014, negotiated timeout = 40000 2018-08-27 15:11:05,025 INFO [pool-1-thread-3-SendThread(127.0.0.1:2183)] zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2183, sessionid = 0x3000eb98adc000e, negotiated timeout = 40000 2018-08-27 15:11:05,030 INFO [pool-1-thread-3] tool.Canary: Read from zNode:/hbase on zookeeper instance:127.0.0.1:2183 in 3ms 2018-08-27 15:11:05,030 INFO [pool-1-thread-1] tool.Canary: Read from zNode:/hbase on zookeeper instance:127.0.0.1:2181 in 4ms 2018-08-27 15:11:05,030 INFO [pool-1-thread-2] tool.Canary: Read from zNode:/hbase on zookeeper instance:127.0.0.1:2182 in 4ms 2018-08-27 15:11:05,041 INFO [pool-1-thread-3] zookeeper.ZooKeeper: Session: 0x3000eb98adc000e closed 2018-08-27 15:11:05,041 INFO [pool-1-thread-3-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x3000eb98adc000e 2018-08-27 15:11:05,048 INFO [pool-1-thread-2] zookeeper.ZooKeeper: Session: 0x2000eb86ebb0014 closed 2018-08-27 15:11:05,048 INFO [pool-1-thread-2-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x2000eb86ebb0014 2018-08-27 15:11:05,049 INFO [pool-1-thread-1] zookeeper.ZooKeeper: Session: 0x1000eb86e96000f closed 2018-08-27 15:11:05,049 INFO [pool-1-thread-1-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x1000eb86e96000f 2018-08-27 15:11:06,005 INFO [main] zookeeper.ReadOnlyZKClient: Close zookeeper connection 0x78b729e6 to 127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183 3-NODE ZK ENSEMBLE, 1 NODE DOWN, 1 PERMITTED FAILURE, CANARY SUCCESS :~/work/src/github.com/hbase$ ~/Applications/hbase/bin/hbase org.apache.hadoop.hbase.tool.Canary -zookeeper -treatFailureAsError -permittedZookeeperFailures 1 2018-08-27 15:11:32,381 INFO [main] tool.Canary: Number of execution threads 16 2018-08-27 15:11:32,422 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-08-27 15:11:32,656 INFO [main] zookeeper.ReadOnlyZKClient: Connect 0x78b729e6 to 127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-08-27 15:11:32,669 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT 2018-08-27 15:11:32,669 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_181 2018-08-27 15:11:32,669 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2018-08-27 15:11:32,669 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6] zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre 2018-08-27 15:11:32,669 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2018-08-27 15:11:32,669 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6] zookeeper.ZooKeeper: Client environment:java.compiler= 2018-08-27 15:11:32,669 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6] zookeeper.ZooKeeper: Client environment:os.name=Linux 2018-08-27 15:11:32,669 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6] zookeeper.ZooKeeper: Client environment:os.arch=amd64 2018-08-27 15:11:32,670 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6] zookeeper.ZooKeeper: Client environment:os.version=4.4.0-134-generic 2018-08-27 15:11:32,672 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6] zookeeper.ZooKeeper: Initiating client connection, connectString=127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$13/788386517@5022a1e1 2018-08-27 15:11:32,693 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6-SendThread(127.0.0.1:2182)] zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2182. Will not attempt to authenticate using SASL (unknown error) 2018-08-27 15:11:32,703 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6-SendThread(127.0.0.1:2182)] zookeeper.ClientCnxn: Socket connection established to 127.0.0.1/127.0.0.1:2182, initiating session 2018-08-27 15:11:32,724 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6-SendThread(127.0.0.1:2182)] zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2182, sessionid = 0x2000eb86ebb0015, negotiated timeout = 40000 2018-08-27 15:11:32,737 WARN [main] client.ConnectionImplementation: Retrieve cluster id failed java.util.concurrent.ExecutionException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/hbaseid at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:542) at org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:297) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.client.ConnectionFactory.lambda$createConnection$0(ConnectionFactory.java:229) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:347) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:227) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:127) at org.apache.hadoop.hbase.tool.Canary.run(Canary.java:798) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1609) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/hbaseid at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:174) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) at java.lang.Thread.run(Thread.java:748) 2018-08-27 15:11:32,856 INFO [pool-1-thread-1] zookeeper.ZooKeeper: Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.EmptyWatcher@1e598625 2018-08-27 15:11:32,856 INFO [pool-1-thread-3] zookeeper.ZooKeeper: Initiating client connection, connectString=127.0.0.1:2183 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.EmptyWatcher@1e598625 2018-08-27 15:11:32,856 INFO [pool-1-thread-2] zookeeper.ZooKeeper: Initiating client connection, connectString=127.0.0.1:2182 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.EmptyWatcher@1e598625 2018-08-27 15:11:32,857 INFO [pool-1-thread-1-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2018-08-27 15:11:32,858 INFO [pool-1-thread-1-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session 2018-08-27 15:11:32,858 INFO [pool-1-thread-2-SendThread(127.0.0.1:2182)] zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2182. Will not attempt to authenticate using SASL (unknown error) 2018-08-27 15:11:32,858 INFO [pool-1-thread-3-SendThread(127.0.0.1:2183)] zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2183. Will not attempt to authenticate using SASL (unknown error) 2018-08-27 15:11:32,859 INFO [pool-1-thread-3-SendThread(127.0.0.1:2183)] zookeeper.ClientCnxn: Socket connection established to 127.0.0.1/127.0.0.1:2183, initiating session 2018-08-27 15:11:32,859 INFO [pool-1-thread-2-SendThread(127.0.0.1:2182)] zookeeper.ClientCnxn: Socket connection established to 127.0.0.1/127.0.0.1:2182, initiating session 2018-08-27 15:11:32,870 INFO [pool-1-thread-3-SendThread(127.0.0.1:2183)] zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2183, sessionid = 0x3000eb98adc000f, negotiated timeout = 40000 2018-08-27 15:11:32,870 INFO [pool-1-thread-2-SendThread(127.0.0.1:2182)] zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2182, sessionid = 0x2000eb86ebb0016, negotiated timeout = 40000 2018-08-27 15:11:32,880 INFO [pool-1-thread-2] tool.Canary: Read from zNode:/hbase on zookeeper instance:127.0.0.1:2182 in 5ms 2018-08-27 15:11:32,880 INFO [pool-1-thread-3] tool.Canary: Read from zNode:/hbase on zookeeper instance:127.0.0.1:2183 in 5ms 2018-08-27 15:11:32,898 INFO [pool-1-thread-2] zookeeper.ZooKeeper: Session: 0x2000eb86ebb0016 closed 2018-08-27 15:11:32,898 INFO [pool-1-thread-2-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x2000eb86ebb0016 2018-08-27 15:11:32,904 INFO [pool-1-thread-3] zookeeper.ZooKeeper: Session: 0x3000eb98adc000f closed 2018-08-27 15:11:32,904 INFO [pool-1-thread-3-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x3000eb98adc000f 2018-08-27 15:12:32,756 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6] zookeeper.ZooKeeper: Session: 0x2000eb86ebb0015 closed 2018-08-27 15:12:32,756 INFO [ReadOnlyZKClient-127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183@0x78b729e6-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x2000eb86ebb0015 2018-08-27 15:13:02,895 WARN [pool-1-thread-1-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 90037ms for sessionid 0x0 2018-08-27 15:13:02,895 INFO [pool-1-thread-1-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 90037ms for sessionid 0x0, closing socket connection and attempting reconnect 2018-08-27 15:13:02,997 ERROR [pool-1-thread-1] tool.Canary: Read from zNode:/hbase on zookeeper instance:127.0.0.1:2181 org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase 2018-08-27 15:13:03,582 INFO [pool-1-thread-1] zookeeper.ZooKeeper: Session: 0x0 closed 2018-08-27 15:13:03,582 INFO [pool-1-thread-1-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x0 2018-08-27 15:13:03,866 INFO [main] zookeeper.ReadOnlyZKClient: Close zookeeper connection 0x78b729e6 to 127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183