Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16765

Fix curator dependencies for gradle projects using hadoop-minicluster

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The Problem:

      The Kudu unit tests that use the `MiniDFSCluster` are broken due to a guava dependency issue in the `hadoop-minicluster` module.

      java.lang.NoSuchMethodError: com.google.common.util.concurrent.Futures.addCallback(Lcom/google/common/util/concurrent/ListenableFuture;Lcom/google/common/util/concurrent/FutureCallback;)V
              at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker.addResultCachingCallback(ThrottledAsyncChecker.java:167)
              at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker.schedule(ThrottledAsyncChecker.java:156)
              at org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:166)
              at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2794)
              at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2709)
              at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1669)
              at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:911)
              at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:518)
              at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:477)
              at org.apache.kudu.backup.HDFSTestKuduBackupLister.setUp(TestKuduBackupLister.scala:216)
      

      The issue in that change is that even though Guava was excluded from the `curator-client` module, just below that the `curator-framework` module is defined and doesn't exclude Gauva:
      https://github.com/apache/hadoop/blob/fccccc97034b29243a0509633849de55aa734859/hadoop-project/pom.xml#L1391-L1414

      This causes Guava 27.0.1-jre to be pulled in instead of Guava 11.0.2 defined by Hadoop:

      +--- org.apache.hadoop:hadoop-minicluster:3.1.1.7.1.0.0-SNAPSHOT
      |    +--- org.apache.hadoop:hadoop-common:3.1.1.7.1.0.0-SNAPSHOT
      |    |    +--- org.apache.hadoop:hadoop-annotations:3.1.1.7.1.0.0-SNAPSHOT
      |    |    +--- com.google.guava:guava:11.0.2 -> 27.0.1-jre
      
      +--- org.apache.curator:curator-framework:4.2.0
      |    \--- org.apache.curator:curator-client:4.2.0
      |         +--- org.apache.zookeeper:zookeeper:3.5.4-beta -> 3.5.5.7.1.0.0-SNAPSHOT (*)
      |         +--- com.google.guava:guava:27.0.1-jre (*)
      |         \--- org.slf4j:slf4j-api:1.7.25

       

      The root cause:

      I was able to reproduce this issue with some dummy projects, see https://github.com/symat/transitive-dependency-test

      It seems that gradle behaves in this case differently than maven. If someone is using maven, then he will not see this problem, as the exclude rules defined for the curator-client will be enforced even if the curator-client comes transitively through the curator-framework. While using the hadoop-minicluster in a gradle project will lead to this problem (unless extra excludes / dependencies gets defined in the gradle project).

      The proposed solution is to add the exclude rules for all Curator dependencies, preventing other gradle projects using Hadoop from breaking because of the Curator upgrade.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                symat Mate Szalay-Beko
                Reporter:
                symat Mate Szalay-Beko
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: