[HADOOP-18709] Add curator based ZooKeeper communication support over SSL/TLS into the common library

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: common
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Ferenc Erdelyi
• Reporter: Ferenc Erdelyi
• Created: 19/Apr/23 15:00
• Updated: 27/Jan/24 08:25
• Resolved: 04/Jun/23 18:41

Description

With HADOOP-16579 the ZooKeeper client is capable of securing communication with SSL. 

To follow the convention introduced in HADOOP-14741, proposing to add to the core-default.xml the following configurations, as the groundwork for the components to enable encrypted communication between the individual components and ZooKeeper:

• hadoop.zk.ssl.keystore.location
• hadoop.zk.ssl.keystore.password
• hadoop.zk.ssl.truststore.location
• hadoop.zk.ssl.truststore.password

These parameters along with the component-specific ssl.client.enable option (e.g. yarn.zookeeper.ssl.client.enable) should be passed to the ZKCuratorManager to build the CuratorFramework. The ZKCuratorManager needs a new overloaded start() method to build the encrypted communication.

• The secured ZK Client uses Netty, hence the dependency is included in the pom.xml. Added netty-handler and netty-transport-native-epoll dependency to the pom.xml based on ZOOKEEPER-3494 - "No need to depend on netty-all (SSL)".
• The change was exclusively tested with the unit test, which is a kind of integration test, as a ZK Server was brought up and the communication tested between the client and the server.
• This code change is in the common code base and there is no component calling it yet. Once YARN-11468 - "Zookeeper SSL/TLS support" is implemented, we can test it in a real cluster environment.

Issue Links

• blocks: YARN-11468 Zookeeper SSL/TLS support
• is broken by: HADOOP-18870 CURATOR-599 change broke functionality introduced in HADOOP-18139 and HADOOP-18709
• relates to: HADOOP-16579 Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.6 in Hadoop
• links to: GitHub Pull Request #5638