[ZOOKEEPER-2541] We can't configure the "secureClientPort" in dynamicConfiguration and connect through client

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.5.1
• Component/s: server
• Assignee: Unassigned
• Reporter: Rakesh Kumar Singh
• Created: 30/Aug/16 05:58
• Updated: 30/Aug/16 05:58

Description

We can't configure the "secureClientPort" in dynamicConfiguration and connect through client

Steps to reproduce:-
1. Configure the zookeeper in cluster mode with SSL mode
2. comment the clientport and secureClientport details from zoo.cfg file
3. Configure the secureClientport in dynamicConfiguration as below:-

server.1=10.18.101.80:2888:3888:participant;2181
server.2=10.18.219.50:2888:3888:participant;2181
server.3=10.18.221.194:2888:3888:participant;2181

4. Start the cluster
5. Start one client using zkCli.sh and try to connect to any one of the cluster, it fails

Client log:-
BLR1000007042:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin # ./zkCli.sh
/usr/bin/java
Connecting to localhost:2181
2016-08-30 13:42:33,574 [myid:] - INFO [main:Environment@109] - Client environment:zookeeper.version=3.5.1-alpha--1, built on 08/18/2016 08:20 GMT
2016-08-30 13:42:33,578 [myid:] - INFO [main:Environment@109] - Client environment:host.name=BLR1000007042
2016-08-30 13:42:33,578 [myid:] - INFO [main:Environment@109] - Client environment:java.version=1.7.0_79
2016-08-30 13:42:33,581 [myid:] - INFO [main:Environment@109] - Client environment:java.vendor=Oracle Corporation
2016-08-30 13:42:33,581 [myid:] - INFO [main:Environment@109] - Client environment:java.home=/usr/java/jdk1.7.0_79/jre
2016-08-30 13:42:33,581 [myid:] - INFO [main:Environment@109] - Client environment:java.class.path=/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../build/classes:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../build/lib/*.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/slf4j-log4j12-1.7.5.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/slf4j-api-1.7.5.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/servlet-api-2.5-20081211.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/netty-3.7.0.Final.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/log4j-1.2.16.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jline-2.11.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jetty-util-6.1.26.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jetty-6.1.26.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/javacc.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jackson-mapper-asl-1.9.11.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/jackson-core-asl-1.9.11.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/commons-cli-1.2.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../lib/ant-eclipse-1.0-jvm1.2.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../zookeeper-3.5.1-alpha.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../src/java/lib/ant-eclipse-1.0-jvm1.2.jar:/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin/../conf:
2016-08-30 13:42:33,582 [myid:] - INFO [main:Environment@109] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-08-30 13:42:33,582 [myid:] - INFO [main:Environment@109] - Client environment:java.io.tmpdir=/tmp
2016-08-30 13:42:33,582 [myid:] - INFO [main:Environment@109] - Client environment:java.compiler=<NA>
2016-08-30 13:42:33,582 [myid:] - INFO [main:Environment@109] - Client environment:os.name=Linux
2016-08-30 13:42:33,582 [myid:] - INFO [main:Environment@109] - Client environment:os.arch=amd64
2016-08-30 13:42:33,583 [myid:] - INFO [main:Environment@109] - Client environment:os.version=3.0.76-0.11-default
2016-08-30 13:42:33,583 [myid:] - INFO [main:Environment@109] - Client environment:user.name=root
2016-08-30 13:42:33,583 [myid:] - INFO [main:Environment@109] - Client environment:user.home=/root
2016-08-30 13:42:33,583 [myid:] - INFO [main:Environment@109] - Client environment:user.dir=/home/Rakesh/Zookeeper/18_Aug/cluster/zookeeper-3.5.1-alpha/bin
2016-08-30 13:42:33,583 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.free=52MB
2016-08-30 13:42:33,586 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.max=227MB
2016-08-30 13:42:33,587 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.total=57MB
2016-08-30 13:42:33,591 [myid:] - INFO [main:ZooKeeper@716] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@272f15b0
Welcome to ZooKeeper!
2016-08-30 13:42:33,681 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1138] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
[INFO] Unable to bind key for unsupported operation: backward-delete-word
[INFO] Unable to bind key for unsupported operation: backward-delete-word
[INFO] Unable to bind key for unsupported operation: down-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: down-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: down-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: down-history
[INFO] Unable to bind key for unsupported operation: up-history
[INFO] Unable to bind key for unsupported operation: down-history
[zk: localhost:2181(CONNECTING) 0] 2016-08-30 13:42:33,975 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxnSocketNetty$ZKClientPipelineFactory@363] - SSL handler added for channel: null
2016-08-30 13:42:34,004 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@980] - Socket connection established, initiating session, client: /0:0:0:0:0:0:0:1:47374, server: localhost/0:0:0:0:0:0:0:1:2181
2016-08-30 13:42:34,006 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxnSocketNetty$1@146] - channel is connected: [id: 0xd4aaee7b, /0:0:0:0:0:0:0:1:47374 => localhost/0:0:0:0:0:0:0:1:2181]
2016-08-30 13:42:34,030 [myid:] - INFO [New I/O worker #1:ClientCnxnSocketNetty$ZKClientHandler@377] - channel is disconnected: [id: 0xd4aaee7b, /0:0:0:0:0:0:0:1:47374 :> localhost/0:0:0:0:0:0:0:1:2181]
2016-08-30 13:42:34,030 [myid:] - INFO [New I/O worker #1:ClientCnxnSocketNetty@201] - channel is told closing
2016-08-30 13:42:34,030 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1253] - channel for sessionid 0x0 is lost, closing socket connection and attempting reconnect
2016-08-30 13:42:34,033 [myid:] - WARN [New I/O worker #1:ClientCnxnSocketNetty$ZKClientHandler@432] - Exception caught: [id: 0xd4aaee7b, /0:0:0:0:0:0:0:1:47374 :> localhost/0:0:0:0:0:0:0:1:2181] EXCEPTION: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at org.jboss.netty.handler.ssl.SslHandler$6.run(SslHandler.java:1580)
at org.jboss.netty.channel.socket.ChannelRunnableWrapper.run(ChannelRunnableWrapper.java:40)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:71)
at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:57)
at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.jboss.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
at org.jboss.netty.handler.ssl.SslHandler.channelClosed(SslHandler.java:1566)
at org.jboss.netty.channel.Channels.fireChannelClosed(Channels.java:468)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:376)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-08-30 13:42:34,230 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1138] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-08-30 13:42:34,240 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxnSocketNetty$ZKClientPipelineFactory@363] - SSL handler added for channel: null
2016-08-30 13:42:34,241 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@980] - Socket connection established, initiating session, client: /127.0.0.1:60295, server: localhost/127.0.0.1:2181

Server log (It is not starting in secureMode though all the required configuration is done for ssl except secureClientport which is configured in dynamicConfiguration):-
2016-08-30 13:40:13,436 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@928] - Notification time out: 800
2016-08-30 13:40:14,239 [myid:1] - WARN [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):QuorumCnxManager@459] - Cannot open channel to 2 at election address /10.18.219.50:3888
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:444)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:485)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:513)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:919)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1040)
201