[SPARK-35954] [Deploy] Upgrade Apache Curator Dependency to 4.2.0

• Type: Dependency upgrade
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.2
• Component/s: Deploy

Environment

• OS: Linux
• JAVA: 1.8.0_292
• hadoop-3.3.1

 

• Assignee: Unassigned
• Reporter: Nikita ROUSSEAU
• Created: 30/Jun/21 15:25
• Updated: 30/Jun/21 16:00

Description

Abstract : as a Spark Cluster Administrator, I want to connect spark masters deployed in HA/mode to Zookeeper over SSL/TLS, so that my network traffic is ciphered between y components. (https://spark.apache.org/docs/latest/spark-standalone.html#standby-masters-with-zookeeper  )

 

With the release of Hadoop 3.3.1, ZKFC libraries and their dependencies were updated : it is now possible to connect ZKFC to ZooKeeper over TLS.

 

Note: TLS is possible with Zookeeper Server Version >= 3.5.6 (https://docs.confluent.io/platform/current/installation/versions-interoperability.html#zk )

 

Spark 3.2.0 aims to support Hadoop 3.3.1 ; this Hadoop release bundles the following shared libraries :

• curator-client-4.2.0.jar
• curator-framework-4.2.0.jar
• curator-recipes-4.2.0.jar
• zookeeper-3.5.6.jar
• zookeeper-jute-3.5.6.jar

 

Currently, Spark dependency is set to 2.13.0 for the Currator framework (https://github.com/apache/spark/blob/master/pom.xml#L127).

 

It would be great to update "curator-*" dependencies to 4.2.0 in order to be compatible with shared jars of the hadoop stack.

Moreover, it will allow administrators to connect Spark Masters to ZooKeeper over TLS.

 

Some patches will be required, such as :

• https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkCuratorUtil.scala#L51

 

I will try to prepare a MR for this.

 

 

Issue Links

• links to: Github PR 33157 - Upgrade Apache Curator Dependency to 4.2.0