[HADOOP-16765] Fix curator dependencies for gradle projects using hadoop-minicluster

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.0
• Hadoop Flags: Reviewed
• Assignee: Mate Szalay-Beko
• Reporter: Mate Szalay-Beko
• Created: 16/Dec/19 08:15
• Updated: 16/Dec/19 22:59
• Resolved: 16/Dec/19 22:45

Description

The Problem:

The Kudu unit tests that use the `MiniDFSCluster` are broken due to a guava dependency issue in the `hadoop-minicluster` module.

java.lang.NoSuchMethodError: com.google.common.util.concurrent.Futures.addCallback(Lcom/google/common/util/concurrent/ListenableFuture;Lcom/google/common/util/concurrent/FutureCallback;)V
        at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker.addResultCachingCallback(ThrottledAsyncChecker.java:167)
        at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker.schedule(ThrottledAsyncChecker.java:156)
        at org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:166)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2794)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2709)
        at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1669)
        at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:911)
        at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:518)
        at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:477)
        at org.apache.kudu.backup.HDFSTestKuduBackupLister.setUp(TestKuduBackupLister.scala:216)

The issue in that change is that even though Guava was excluded from the `curator-client` module, just below that the `curator-framework` module is defined and doesn't exclude Gauva:
https://github.com/apache/hadoop/blob/fccccc97034b29243a0509633849de55aa734859/hadoop-project/pom.xml#L1391-L1414

This causes Guava 27.0.1-jre to be pulled in instead of Guava 11.0.2 defined by Hadoop:

+--- org.apache.hadoop:hadoop-minicluster:3.1.1.7.1.0.0-SNAPSHOT
|    +--- org.apache.hadoop:hadoop-common:3.1.1.7.1.0.0-SNAPSHOT
|    |    +--- org.apache.hadoop:hadoop-annotations:3.1.1.7.1.0.0-SNAPSHOT
|    |    +--- com.google.guava:guava:11.0.2 -> 27.0.1-jre

+--- org.apache.curator:curator-framework:4.2.0
|    \--- org.apache.curator:curator-client:4.2.0
|         +--- org.apache.zookeeper:zookeeper:3.5.4-beta -> 3.5.5.7.1.0.0-SNAPSHOT (*)
|         +--- com.google.guava:guava:27.0.1-jre (*)
|         \--- org.slf4j:slf4j-api:1.7.25

 

The root cause:

I was able to reproduce this issue with some dummy projects, see https://github.com/symat/transitive-dependency-test

It seems that gradle behaves in this case differently than maven. If someone is using maven, then he will not see this problem, as the exclude rules defined for the curator-client will be enforced even if the curator-client comes transitively through the curator-framework. While using the hadoop-minicluster in a gradle project will lead to this problem (unless extra excludes / dependencies gets defined in the gradle project).

The proposed solution is to add the exclude rules for all Curator dependencies, preventing other gradle projects using Hadoop from breaking because of the Curator upgrade.

Issue Links

• links to: GitHub Pull Request #1763