+1

Flavio Junqueira
That sounds interesting!
What about using SSLEngine

these are related issues which are quite old

I second the ssl-engine idea Hongchao Deng suggested.

Is it possible to use Netty+SSL?

Since netty support for client-to-server communication is already in place, adding netty ssl to it would be a nice idea.

We added netty in the first place primarily to enable adding ssl support.

Netty+SSL sounds a great idea. It might force people to start using netty. Good or bad though.

ZK didn't have netty support for server-to-server communication. If anyone is interested, this might be a good chance to support that.

Hi Michi Mutsuzaki. Did you add SSL support on the zab project?? Might be a good case to borrow from.

Yes, we used netty and added ssl support:

https://github.com/zk1931/jzab/blob/master/src/main/java/com/github/zk1931/jzab/transport/NettyTransport.java
https://github.com/zk1931/jzab/blob/0ab137d0650b03425df7840972eb9de4ba123030/src/test/java/com/github/zk1931/jzab/transport/NettyTransportTest.java#L519

Right now ZooKeeper supports netty only for client-to-server communication on the server side (NettyServerCnxn.java). Server-to-server communication uses java.net.Socket in Leader.java, PeerHandler.java, Learner.java, and QuorumCnxManager.java. Maybe this issue can be broken into smaller subtasks:

• implement a netty version of ClientCnxnSocket.
• modify Leader.java/PeerHandler.java to use netty.
• modify Learner.java to use netty.
• modify QuorumCnxManager.java to use netty.

Ideally we should have "netty client socket" and "netty server socket" classes that all of these classes can reuse so that we don't repeat logic for netty pipeline initialization and things.

In which version will the above change be available?

Looks like netty+ssl gains more votes here. It's a good place to start supporting SSL because it encapsulates those handshake details.

I will work on netty+ssl since I have seen the needs of a couple of users and community. Anyone who's interested can ping me as well.

Hi, Hongchao,

Will this feature be available in 3.5.0 or 3.5.1 ?

3.5.0 has been released. Hopefully in 3.5.1

Got it, Thanks a lot

I could use something like zookeeper. Without the needed SSL support on all communications, it is unusable to me.

Hi Hongchao Deng,

I noticed ZOOKEEPER-2125 which addresses the client-server SSL in 3.5.1. Will server-server be addressed as well?

Thanks,
Shane

Yes.

I'm currently working on server-server as well as client-server which can be backported onto 3.4 branch. It took some time though.

Hello Hongchao!
I am trying to test SSL connectivity between client and server following the instructions in the following page:
https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide

But, I get the following when trying to connect to server from client..

2015-06-24 12:14:36,589 [myid:] - INFO [main:ZooKeeper@709] - Initiating client connection, connectString=localhost:2282 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@f2a0b8e
Exception in thread "main" java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNetty

Can you tell me what I am doing wrong here?
Very new to Zookeeper.
Thanks!

Can you open a new JIRA?

Can you open a new JIRA?

Hongchao Deng Wondering if you are still working on this, what the time frame might be? If not, is there some work in progress that another could take up and work on?

Moving discussion from ZOOKEEPER-1045.
I have a patch in the works for QuorumCnxManager using Netty + SSL. I initially intended to use Netty for Learner/Leader but that now seems more daunting. Hence I have decided to get this in two steps. SSLSockets for Learner/Leader and Netty + SSL for QuorumCnxManager.

I do have something that implements SSLSockets for Learner/Leader and did the same for QCM just so that I can hopefully get more feedback and insight into SSL support requirement while I wrap up Netty + SSL for QCM.

Please find the repo and readme here:
https://github.com/geek101/zookeeper/blob/branch-3.4/README_SSL.md

Code changes and scripts:
https://github.com/geek101/zookeeper/commit/0ff011ed42513491af90277cd251c78f4a8f3e36

All feedback is welcome.

Powell Molleti when you're ready, could you actually run some performance tests to make sure this isn't causing a perf hit?

Updated the read-me with benchmark. Posting summary here:

A 3 Server ensemble running in a single VM, command used(single client)
zk-latencies.py -cluster "127.0.1.1:2181,127.0.1.2:2181,127.0.1.3:2181"-znode_size=100 --znode_count=10000 --timeout=5000 --watch_multiple=5

SSL Disabled:
110000 ops took 16441.6 ms

SSL Enabled:
110000 ops took 16721.2 ms

From my understanding SSL support will have measurable impact in connection setup latency hence CPS bound application will suffer. Due to widespread AES-NI support on CPUs (and Java supports this) I am not quite certain w.r.t latency added for steady state data transfer's encrypt/decrypt process, potentially impacting ZAB. I was hoping that any such measurable latency might show up in smoke tests which were run the past. Let me know if you other perf tests in mind that I can focus on next to get more insight.

I am running into limits of single VM, I would like to run with more clients to replicate past tests

I would like to start using a ZooKeeper over a WAN network using SSL and willing to contribute in terms of code and testing. Is there anyone who could direct me to the latest branch with the SSL-enabled code?

Hi Andrey,

SSL for FLE and ZAB right? and is it for branch 3.4? then I just rebased(not tested) to latest on 3.4 and squashed all the commits to one. You can look at instructions here:
https://github.com/geek101/zookeeper/blob/branch-3.4/README_SSL.md

For 3.5 (to support dynamic reconfig with SSL along with fault tolerance) it is more involved and you can look at the code here:
https://github.com/geek101/zookeeper/pull/5/files

The above pull request is work in progress and not ready for general consumption yet.

thanks
Powell.

We've been using this branch now in production for some time now, it's working well. I did have to extend it a bit with this PR: https://github.com/geek101/zookeeper/pull/1
which makes it a bit more generic in the configuration of the certs, etc.

I'd love to see this upstreamed for 3.4.X (and eventually 3.5.X)

Jason Rosenberg Great to hear that you're using it successfully in production. The plan so far is to have it in 3.5 and trunk, not in 3.4 (see fix version above).

Hi Flavio Junqueira since 3.5.X is still alpha, it would of course be great to get this out under 3.4.X as well!

Also, for 3.5.X, can we see about also including the mods I referenced above as well, in order to support a more general configuration for certs (e.g. different file formats/file locations)?