Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.6.0
    • Fix Version/s: None
    • Component/s: build
    • Labels:
      None

      Description

      HADOOP-10868 includes apache curator 2.6.0

      This depends on Guava 16.01

      It's not being picked up, as Hadoop is forcing in 11.0.2 -but this means:
      there is now a risk that curator depends on methods and classes that are not in the Hadoop version

      1. HADOOP-11102-001.patch
        3 kB
        Steve Loughran
      2. HADOOP-11102-002.patch
        3 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          I need curator for YARN-913; I'd been sticking to 2.4.1 as it's jackson version was compatible with Hadoop 2.4, and it was build against Guava 14.0.1.

          Importing curator 2.6 means there's a significant risk that its eventually going to call classes and methods that arent in the 11.02 JAR

          Where does that leave Hadoop?

          1. do the guava update? I'm personally in favour of this. See HADOOP-10101 patch for that
          2. make sure that Hadoop can run against Guava 16, which means moving off anything deprecated in the current builds, as well as working around other incompatible changes (e.g. HADOOP-10961)

          What we can't do is ignore this as we now have an inconsistent set of dependencies

          Show
          stevel@apache.org Steve Loughran added a comment - I need curator for YARN-913 ; I'd been sticking to 2.4.1 as it's jackson version was compatible with Hadoop 2.4, and it was build against Guava 14.0.1. Importing curator 2.6 means there's a significant risk that its eventually going to call classes and methods that arent in the 11.02 JAR Where does that leave Hadoop? do the guava update? I'm personally in favour of this. See HADOOP-10101 patch for that make sure that Hadoop can run against Guava 16, which means moving off anything deprecated in the current builds, as well as working around other incompatible changes (e.g. HADOOP-10961 ) What we can't do is ignore this as we now have an inconsistent set of dependencies
          Hide
          stevel@apache.org Steve Loughran added a comment -

          My plan: stay with guava 11 in hadoop 2.x, fix things to work there and against later versions of Guava —so people can update if they want to, even though hadoop doesn't itself.

          1. fix Hadoop so it builds/runs against Guava 16 as an option. That means: replace classes used in Hadoop which are not in Guava 16 with alternatives (as HBase has done)
          2. add a -Pguava-latest profile to verify hadoop does build against it
          3. roll back to Curator 2.4.1, which only depends on guava 14. AFAIK there are no classes in guava 11 that aren't in guava 14.

          Hadoop 3 can move to guava 16+

          Show
          stevel@apache.org Steve Loughran added a comment - My plan: stay with guava 11 in hadoop 2.x, fix things to work there and against later versions of Guava —so people can update if they want to, even though hadoop doesn't itself. fix Hadoop so it builds/runs against Guava 16 as an option . That means: replace classes used in Hadoop which are not in Guava 16 with alternatives (as HBase has done) add a -Pguava-latest profile to verify hadoop does build against it roll back to Curator 2.4.1, which only depends on guava 14. AFAIK there are no classes in guava 11 that aren't in guava 14. Hadoop 3 can move to guava 16+
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 001: roll back curator to 2.4.1 and hence guava to 14, zk to 3.4.5. changes pom.xml and TestZKDelegationTokenSecretManager

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 001: roll back curator to 2.4.1 and hence guava to 14, zk to 3.4.5. changes pom.xml and TestZKDelegationTokenSecretManager
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12674563/HADOOP-11102-001.patch
          against trunk revision a56ea01.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4912//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12674563/HADOOP-11102-001.patch against trunk revision a56ea01. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4912//console This message is automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          cancelling ... patch was against branch-2 & clearly not applying to trunk

          Show
          stevel@apache.org Steve Loughran added a comment - cancelling ... patch was against branch-2 & clearly not applying to trunk
          Hide
          stevel@apache.org Steve Loughran added a comment -

          patch that applies to trunk. The POM changes also cull the unwanted duplicate declarations of curator as a dependency,.

          Show
          stevel@apache.org Steve Loughran added a comment - patch that applies to trunk. The POM changes also cull the unwanted duplicate declarations of curator as a dependency,.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12674591/HADOOP-11102-002.patch
          against trunk revision a56ea01.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4914//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12674591/HADOOP-11102-002.patch against trunk revision a56ea01. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4914//console This message is automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I did two test builds of curator with guava set to 11.0.2. Curator 2.4.1 builds; Curator 2.6.0 fails in curator-x-discovery-server:

          [ERROR] /Users/stevel/Projects/Hortonworks/Projects/curator/curator-x-discovery-server/src/main/java/org/apache/curator/x/discovery/server/contexts/GenericDiscoveryContext.java:[21,33] package com.google.common.reflect does not exist
          [ERROR] /Users/stevel/Projects/Hortonworks/Projects/curator/curator-x-discovery-server/src/main/java/org/apache/curator/x/discovery/server/contexts/GenericDiscoveryContext.java:[40,19] cannot find symbol
          [ERROR] symbol:   class TypeToken
          [ERROR] location: class org.apache.curator.x.discovery.server.contexts.GenericDiscoveryContext<T>
          [ERROR] /Users/stevel/Projects/Hortonworks/Projects/curator/curator-x-discovery-server/src/main/java/org/apache/curator/x/discovery/server/contexts/GenericDiscoveryContext.java:[47,135] cannot find symbol
          [ERROR] symbol:   class TypeToken
          [ERROR] location: class org.apache.curator.x.discovery.server.contexts.GenericDiscoveryContext<T>
          [ERROR] /Users/stevel/Projects/Hortonworks/Projects/curator/curator-x-discovery-server/src/main/java/org/apache/curator/x/discovery/server/contexts/MapDiscoveryContext.java:[21,33] package com.google.common.reflect does not exist
          [ERROR] /Users/stevel/Projects/Hortonworks/Projects/curator/curator-x-discovery-server/src/main/java/org/apache/curator/x/discovery/server/contexts/GenericDiscoveryContext.java:[44,69] cannot find symbol
          [ERROR] symbol:   variable TypeToken
          [ERROR] location: class org.apache.curator.x.discovery.server.contexts.GenericDiscoveryContext<T>
          [ERROR] /Users/stevel/Projects/Hortonworks/Projects/curator/curator-x-discovery-server/src/main/java/org/apache/curator/x/discovery/server/contexts/GenericDiscoveryContext.java:[42,12] recursive constructor invocation
          [ERROR] /Users/stevel/Projects/Hortonworks/Projects/curator/curator-x-discovery-server/src/main/java/org/apache/curator/x/discovery/server/contexts/MapDiscoveryContext.java:[38,74] cannot find symbol
          [ERROR] symbol:   class TypeToken
          [ERROR] location: class org.apache.curator.x.discovery.server.contexts.MapDiscoveryContext
          [ERROR] -> [Help 1]
          

          This implies that part of curator is not compatible with guava 11, though it is not a core module.
          As 2.4.1 builds without problems, I think the patch should still go in

          Show
          stevel@apache.org Steve Loughran added a comment - I did two test builds of curator with guava set to 11.0.2. Curator 2.4.1 builds; Curator 2.6.0 fails in curator-x-discovery-server : [ERROR] /Users/stevel/Projects/Hortonworks/Projects/curator/curator-x-discovery-server/src/main/java/org/apache/curator/x/discovery/server/contexts/GenericDiscoveryContext.java:[21,33] package com.google.common.reflect does not exist [ERROR] /Users/stevel/Projects/Hortonworks/Projects/curator/curator-x-discovery-server/src/main/java/org/apache/curator/x/discovery/server/contexts/GenericDiscoveryContext.java:[40,19] cannot find symbol [ERROR] symbol: class TypeToken [ERROR] location: class org.apache.curator.x.discovery.server.contexts.GenericDiscoveryContext<T> [ERROR] /Users/stevel/Projects/Hortonworks/Projects/curator/curator-x-discovery-server/src/main/java/org/apache/curator/x/discovery/server/contexts/GenericDiscoveryContext.java:[47,135] cannot find symbol [ERROR] symbol: class TypeToken [ERROR] location: class org.apache.curator.x.discovery.server.contexts.GenericDiscoveryContext<T> [ERROR] /Users/stevel/Projects/Hortonworks/Projects/curator/curator-x-discovery-server/src/main/java/org/apache/curator/x/discovery/server/contexts/MapDiscoveryContext.java:[21,33] package com.google.common.reflect does not exist [ERROR] /Users/stevel/Projects/Hortonworks/Projects/curator/curator-x-discovery-server/src/main/java/org/apache/curator/x/discovery/server/contexts/GenericDiscoveryContext.java:[44,69] cannot find symbol [ERROR] symbol: variable TypeToken [ERROR] location: class org.apache.curator.x.discovery.server.contexts.GenericDiscoveryContext<T> [ERROR] /Users/stevel/Projects/Hortonworks/Projects/curator/curator-x-discovery-server/src/main/java/org/apache/curator/x/discovery/server/contexts/GenericDiscoveryContext.java:[42,12] recursive constructor invocation [ERROR] /Users/stevel/Projects/Hortonworks/Projects/curator/curator-x-discovery-server/src/main/java/org/apache/curator/x/discovery/server/contexts/MapDiscoveryContext.java:[38,74] cannot find symbol [ERROR] symbol: class TypeToken [ERROR] location: class org.apache.curator.x.discovery.server.contexts.MapDiscoveryContext [ERROR] -> [Help 1] This implies that part of curator is not compatible with guava 11, though it is not a core module. As 2.4.1 builds without problems, I think the patch should still go in
          Hide
          rkanter Robert Kanter added a comment -

          +1 (non-binding)

          The curator-x-discovery-server is for the Service Discovery Server, which creates a RESTful server interface for non-Java programs to use the Service Discovery recipe (curator-x-discovery), and I don't think Hadoop is currently using either of those, so it should be fine; at least for now.

          Show
          rkanter Robert Kanter added a comment - +1 (non-binding) The curator-x-discovery-server is for the Service Discovery Server, which creates a RESTful server interface for non-Java programs to use the Service Discovery recipe ( curator-x-discovery ), and I don't think Hadoop is currently using either of those, so it should be fine; at least for now.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Apparently OOzie and WebHCat works with hadoop 2.x & Curator 2.6 on its classpath, so it is compatible for all main uses other than the x-service-discovery module. Closing as a WONTFIX

          Show
          stevel@apache.org Steve Loughran added a comment - Apparently OOzie and WebHCat works with hadoop 2.x & Curator 2.6 on its classpath, so it is compatible for all main uses other than the x-service-discovery module. Closing as a WONTFIX

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development