Hadoop Common
  1. Hadoop Common
  2. HADOOP-9991

Fix up Hadoop Poms for enforced dependencies, roll up JARs to latest versions

        Details

        • Type: Improvement Improvement
        • Status: Open
        • Priority: Major Major
        • Resolution: Unresolved
        • Affects Version/s: 2.1.1-beta, 2.3.0
        • Fix Version/s: None
        • Component/s: build
        • Labels:
          None
        • Target Version/s:

          Description

          If you try using Hadoop downstream with a classpath shared with HBase and Accumulo, you soon discover how messy the dependencies are.

          Hadoop's side of this problem is

          1. not being up to date with some of the external releases of common JARs
          2. not locking down/excluding inconsistent versions of artifacts provided down the dependency graph

            Issue Links

            depends upon

            Sub-task - The sub-task of the issue HDFS-5411 Update Bookkeeper dependency to 4.2.3

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-9555 HA functionality that uses ZooKeeper may experience inadvertent TCP RST and miss session expiration event due to bug in client connection management

            • Major - Major loss of function.
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-10814 Update Tomcat version used by HttpFS and KMS to latest 6.x version

            • Major - Major loss of function.
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-5431 Missing pom dependency in MR-client

            • Major - Major loss of function.
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-10147 Upgrade to commons-logging 1.1.3 to avoid potential deadlock in MiniDFSCluster

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-11515 Upgrade jsch lib to jsch-0.1.51 to avoid problems running on java7

            • Major - Major loss of function.
            • Patch Available

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-10075 Update jetty dependency to version 9

            • Major - Major loss of function.
            • Open

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-10076 Update tomcat/jasper dependency to version 7

            • Major - Major loss of function.
            • Open

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-9244 Upgrade servlet-api dependency from version 2.5 to 3.0.

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Open

            Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-5624 move grizzly-test and junit dependencies to test scope

            • Major - Major loss of function.
            • Resolved

            Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-5678 Move junit to test scope in more projects

            • Major - Major loss of function.
            • Resolved

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-10384 CLONE - Upgrade servlet-api dependency from version 2.5 to 3.0.

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Resolved

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-9594 Update apache commons math dependency

            • Major - Major loss of function.
            • Closed

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-9611 mvn-rpmbuild against google-guice > 3.0 yields missing cglib dependency

            • Major - Major loss of function.
            • Closed

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-9833 move slf4j to version 1.7.5

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Closed

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-9613 Updated jersey pom dependencies

            • Major - Major loss of function.
            • Patch Available

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-10101 Update guava dependency to the latest version

            • Major - Major loss of function.
            • Patch Available
            is related to

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-10783 apache-commons-lang.jar 2.6 does not support FreeBSD -upgrade to 3.x needed

            • Major - Major loss of function.
            • Open

            Bug - A problem which impairs or prevents the functions of the product. BOOKKEEPER-708 Shade protobuf library to avoid incompatible versions

            • Major - Major loss of function.
            • Resolved

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-10100 MiniKDC shouldn't use apacheds-all artifact

            • Major - Major loss of function.
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-9961 versions of a few transitive dependencies diverged between hadoop subprojects

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-8793 hadoop-core has dependencies on two different versions of commons-httpclient

            • Critical - Crashes, loss of data, severe memory leak.
            • Patch Available

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-9905 remove dependency of zookeeper for hadoop-client

            • Major - Major loss of function.
            • Patch Available

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-10105 remove httpclient dependency

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Open

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-10530 Make hadoop trunk build on Java7+ only

            • Blocker - Blocks development and/or testing work, production could not run
            • Resolved

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-10067 Missing POM dependency on jsr305

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Closed
            relates to

            Task - A task that needs to be done. HADOOP-11492 Bump up curator version to 2.7.1

            • Major - Major loss of function.
            • Resolved

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              Steve Loughran added a comment -

              Proposed Fixes

              1. turn on maven enforcement in the build so highlight inconsistencies coming in from below (e.g Avro's SLF4J dependency)
              2. fix those inconsistencies by excluding the conflicts coming from dependencies
              3. Add explicit imports and scope limits on all dependencies, with version numbers we manage
              4. Tightening the downstream exported dependencies, so hadoop-client only declares dependencies on the JARs it really needs (not, say JUnit).
              5. enum later versions of JARs that we can easily migrate to simply by incrementing JARs: and doing that in trunk, with the enforcer identifying more dependency problems to address.
              6. identifying low-cost updates (ideally those with patches in, like the JetS3t/S3 patch), and selectively applying them -again, fixing problems.

              I'd push this all at trunk, though items 1-4 could be backported to 2.x once complete

              Show
              Steve Loughran added a comment - Proposed Fixes turn on maven enforcement in the build so highlight inconsistencies coming in from below (e.g Avro's SLF4J dependency) fix those inconsistencies by excluding the conflicts coming from dependencies Add explicit imports and scope limits on all dependencies, with version numbers we manage Tightening the downstream exported dependencies, so hadoop-client only declares dependencies on the JARs it really needs (not, say JUnit). enum later versions of JARs that we can easily migrate to simply by incrementing JARs: and doing that in trunk, with the enforcer identifying more dependency problems to address. identifying low-cost updates (ideally those with patches in, like the JetS3t/S3 patch), and selectively applying them -again, fixing problems. I'd push this all at trunk, though items 1-4 could be backported to 2.x once complete
              Hide
              Permalink
              Steve Loughran added a comment -

              enforcement is on, it's more excessive export to things downstream, especially with hbase in the mix -as that is where version number problems start to surface

              This is the main set of "exclusions because they don't appear used" values, though HDFS's JSP pages may well need jasper. The jersey-test-framework-grizzly2 dependency (branch-2.1.1) is clearly spurious

                  <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-minicluster</artifactId>
      <version>${hadoop.version}</version>
      <scope>test</scope>
      <exclusions>
        <exclusion>
          <groupId>com.sun.jersey.jersey-test-framework</groupId>
          <artifactId>jersey-test-framework-grizzly2</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-hdfs</artifactId>
      <version>${hadoop.version}</version>
      <exclusions>
        <exclusion>
          <groupId>tomcat</groupId>
          <artifactId>jasper-runtime</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-yarn-server-common</artifactId>
      <version>${hadoop.version}</version>
      <exclusions>
        <exclusion>
          <groupId>com.sun.jersey.jersey-test-framework</groupId>
          <artifactId>jersey-test-framework-grizzly2</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-yarn-client</artifactId>
      <version>${hadoop.version}</version>
      <exclusions>
        <exclusion>
          <groupId>com.sun.jersey.jersey-test-framework</groupId>
          <artifactId>jersey-test-framework-grizzly2</artifactId>
        </exclusion>
      </exclusions>      
    </dependency>
              Show
              Steve Loughran added a comment - enforcement is on, it's more excessive export to things downstream, especially with hbase in the mix -as that is where version number problems start to surface This is the main set of "exclusions because they don't appear used" values, though HDFS's JSP pages may well need jasper. The jersey-test-framework-grizzly2 dependency (branch-2.1.1) is clearly spurious <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minicluster</artifactId> <version>${hadoop.version}</version> <scope>test</scope> <exclusions> <exclusion> <groupId>com.sun.jersey.jersey-test-framework</groupId> <artifactId>jersey-test-framework-grizzly2</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>${hadoop.version}</version> <exclusions> <exclusion> <groupId>tomcat</groupId> <artifactId>jasper-runtime</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-yarn-server-common</artifactId> <version>${hadoop.version}</version> <exclusions> <exclusion> <groupId>com.sun.jersey.jersey-test-framework</groupId> <artifactId>jersey-test-framework-grizzly2</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-yarn-client</artifactId> <version>${hadoop.version}</version> <exclusions> <exclusion> <groupId>com.sun.jersey.jersey-test-framework</groupId> <artifactId>jersey-test-framework-grizzly2</artifactId> </exclusion> </exclusions> </dependency>
              Hide
              Permalink
              Steve Loughran added a comment -

              MapReduce is marking junit as compile scoped, not test, so trickles into the tarballs and downstream

              Show
              Steve Loughran added a comment - MapReduce is marking junit as compile scoped, not test, so trickles into the tarballs and downstream
              Hide
              Permalink
              Steve Loughran added a comment -

              downstream of the hadoop-client artifact, this is what you get on your dependency graph

              [INFO] \- org.apache.hadoop:hadoop-client:pom:2.1.2-SNAPSHOT:compile
[INFO]    +- org.apache.hadoop:hadoop-common:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  +- commons-cli:commons-cli:jar:1.2:compile
[INFO]    |  +- org.apache.commons:commons-math:jar:2.1:compile
[INFO]    |  +- xmlenc:xmlenc:jar:0.52:compile
[INFO]    |  +- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO]    |  +- commons-io:commons-io:jar:2.1:compile
[INFO]    |  +- commons-logging:commons-logging:jar:1.1.1:compile
[INFO]    |  +- log4j:log4j:jar:1.2.17:compile
[INFO]    |  +- commons-lang:commons-lang:jar:2.5:compile
[INFO]    |  +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO]    |  |  +- commons-collections:commons-collections:jar:3.2.1:compile
[INFO]    |  |  +- commons-digester:commons-digester:jar:1.8:compile
[INFO]    |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO]    |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO]    |  +- org.slf4j:slf4j-log4j12:jar:1.7.5:compile
[INFO]    |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.7:compile
[INFO]    |  +- org.apache.avro:avro:jar:1.7.4:compile
[INFO]    |  |  +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO]    |  |  \- org.xerial.snappy:snappy-java:jar:1.0.4.1:compile
[INFO]    |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO]    |  +- org.apache.hadoop:hadoop-auth:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
[INFO]    |  \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO]    |     \- org.tukaani:xz:jar:1.0:compile
[INFO]    +- org.apache.hadoop:hadoop-hdfs:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  +- com.google.guava:guava:jar:11.0.2:compile
[INFO]    |  |  \- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO]    |  +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO]    |  +- commons-codec:commons-codec:jar:1.4:compile
[INFO]    |  \- org.codehaus.jackson:jackson-core-asl:jar:1.9.7:compile
[INFO]    +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  |  +- org.apache.hadoop:hadoop-yarn-client:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  |  |  +- com.google.inject:guice:jar:3.0:compile
[INFO]    |  |  |  |  +- javax.inject:javax.inject:jar:1:compile
[INFO]    |  |  |  |  \- aopalliance:aopalliance:jar:1.0:compile
[INFO]    |  |  |  +- com.sun.jersey:jersey-server:jar:1.9:compile
[INFO]    |  |  |  |  +- asm:asm:jar:3.1:compile
[INFO]    |  |  |  |  \- com.sun.jersey:jersey-core:jar:1.9:compile
[INFO]    |  |  |  +- com.sun.jersey:jersey-json:jar:1.9:compile
[INFO]    |  |  |  |  +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO]    |  |  |  |  |  \- stax:stax-api:jar:1.0.1:compile
[INFO]    |  |  |  |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO]    |  |  |  |  |  \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
[INFO]    |  |  |  |  |     \- javax.activation:activation:jar:1.1:compile
[INFO]    |  |  |  |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile
[INFO]    |  |  |  |  \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile
[INFO]    |  |  |  \- com.sun.jersey.contribs:jersey-guice:jar:1.9:compile
[INFO]    |  |  \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  +- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  \- org.slf4j:slf4j-api:jar:1.7.5:compile
[INFO]    +- org.apache.hadoop:hadoop-yarn-api:jar:2.1.2-SNAPSHOT:compile
[INFO]    +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.1.2-SNAPSHOT:compile
[INFO]    +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.1.2-SNAPSHOT:compile
[INFO]    \- org.apache.hadoop:hadoop-annotations:jar:2.1.2-SNAPSHOT:compile
              Show
              Steve Loughran added a comment - downstream of the hadoop-client artifact, this is what you get on your dependency graph [INFO] \- org.apache.hadoop:hadoop-client:pom:2.1.2-SNAPSHOT:compile [INFO] +- org.apache.hadoop:hadoop-common:jar:2.1.2-SNAPSHOT:compile [INFO] | +- commons-cli:commons-cli:jar:1.2:compile [INFO] | +- org.apache.commons:commons-math:jar:2.1:compile [INFO] | +- xmlenc:xmlenc:jar:0.52:compile [INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:compile [INFO] | +- commons-io:commons-io:jar:2.1:compile [INFO] | +- commons-logging:commons-logging:jar:1.1.1:compile [INFO] | +- log4j:log4j:jar:1.2.17:compile [INFO] | +- commons-lang:commons-lang:jar:2.5:compile [INFO] | +- commons-configuration:commons-configuration:jar:1.6:compile [INFO] | | +- commons-collections:commons-collections:jar:3.2.1:compile [INFO] | | +- commons-digester:commons-digester:jar:1.8:compile [INFO] | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile [INFO] | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile [INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.5:compile [INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.7:compile [INFO] | +- org.apache.avro:avro:jar:1.7.4:compile [INFO] | | +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile [INFO] | | \- org.xerial.snappy:snappy-java:jar:1.0.4.1:compile [INFO] | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile [INFO] | +- org.apache.hadoop:hadoop-auth:jar:2.1.2-SNAPSHOT:compile [INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile [INFO] | \- org.apache.commons:commons-compress:jar:1.4.1:compile [INFO] | \- org.tukaani:xz:jar:1.0:compile [INFO] +- org.apache.hadoop:hadoop-hdfs:jar:2.1.2-SNAPSHOT:compile [INFO] | +- com.google.guava:guava:jar:11.0.2:compile [INFO] | | \- com.google.code.findbugs:jsr305:jar:1.3.9:compile [INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile [INFO] | +- commons-codec:commons-codec:jar:1.4:compile [INFO] | \- org.codehaus.jackson:jackson-core-asl:jar:1.9.7:compile [INFO] +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.1.2-SNAPSHOT:compile [INFO] | +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.1.2-SNAPSHOT:compile [INFO] | | +- org.apache.hadoop:hadoop-yarn-client:jar:2.1.2-SNAPSHOT:compile [INFO] | | | +- com.google.inject:guice:jar:3.0:compile [INFO] | | | | +- javax.inject:javax.inject:jar:1:compile [INFO] | | | | \- aopalliance:aopalliance:jar:1.0:compile [INFO] | | | +- com.sun.jersey:jersey-server:jar:1.9:compile [INFO] | | | | +- asm:asm:jar:3.1:compile [INFO] | | | | \- com.sun.jersey:jersey-core:jar:1.9:compile [INFO] | | | +- com.sun.jersey:jersey-json:jar:1.9:compile [INFO] | | | | +- org.codehaus.jettison:jettison:jar:1.1:compile [INFO] | | | | | \- stax:stax-api:jar:1.0.1:compile [INFO] | | | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile [INFO] | | | | | \- javax.xml.bind:jaxb-api:jar:2.2.2:compile [INFO] | | | | | \- javax.activation:activation:jar:1.1:compile [INFO] | | | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile [INFO] | | | | \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile [INFO] | | | \- com.sun.jersey.contribs:jersey-guice:jar:1.9:compile [INFO] | | \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.1.2-SNAPSHOT:compile [INFO] | +- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.1.2-SNAPSHOT:compile [INFO] | \- org.slf4j:slf4j-api:jar:1.7.5:compile [INFO] +- org.apache.hadoop:hadoop-yarn-api:jar:2.1.2-SNAPSHOT:compile [INFO] +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.1.2-SNAPSHOT:compile [INFO] | \- org.apache.hadoop:hadoop-yarn-common:jar:2.1.2-SNAPSHOT:compile [INFO] +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.1.2-SNAPSHOT:compile [INFO] \- org.apache.hadoop:hadoop-annotations:jar:2.1.2-SNAPSHOT:compile
              Hide
              Permalink
              Steve Loughran added a comment -
              1. hadoop-hdfs should be excluding com.google.code.findbugs:jsr305:jar:1.3.9 from guava (or downgrade to provided)
              2. hadoop-mapreduce-client-common should not need to depend on yarn-server-common, or jersey-server, though jax
              3. hadoop-common should mark ZK dependency as <provided>, with HDFS server declaring it for HA.
              Show
              Steve Loughran added a comment - hadoop-hdfs should be excluding com.google.code.findbugs:jsr305:jar:1.3.9 from guava (or downgrade to provided) hadoop-mapreduce-client-common should not need to depend on yarn-server-common, or jersey-server, though jax hadoop-common should mark ZK dependency as <provided>, with HDFS server declaring it for HA.
              Hide
              Permalink
              stack added a comment -

              Steve Loughran Thanks for taking on this masochistic task. Given hadoop surface area, it is hard to figure if a dependency is needed. How about we go radical in trunk since it will be ten years before there is a hadoop3 which should be time enough to figure those dependencies removed that should have been left in. hadoop2 could do w/ a just a purge of at least the just-not-used. I'd like to help out. In hbase we put on a few filters to try and block the plain farcical but god-bless-maven, you have to reproduce this set each time for each hadoop version we build against since no xinclude...

              Show
              stack added a comment - Steve Loughran Thanks for taking on this masochistic task. Given hadoop surface area, it is hard to figure if a dependency is needed. How about we go radical in trunk since it will be ten years before there is a hadoop3 which should be time enough to figure those dependencies removed that should have been left in. hadoop2 could do w/ a just a purge of at least the just-not-used. I'd like to help out. In hbase we put on a few filters to try and block the plain farcical but god-bless-maven, you have to reproduce this set each time for each hadoop version we build against since no xinclude...
              Hide
              Permalink
              Steve Loughran added a comment -

              Stack, HBase's needs are one of the things I'm thinking of, and once you get into the goal of including Hadoop, HBase and accumulo into a project, things get worse to the extent I couldn't even pull in htmlunit: https://github.com/hortonworks/hoya/blob/master/pom.xml

              1. We should really add ZK and avro into the mix here too
              2. I'd love to see stripped down clients -which we could address by having some lean poms that only include the core bits to talk to localfs, hdfs, yarn, MR, all optional client-side bits (avro integration, ftpfs, s3fs, pulled to the side where you can add them if you explicitly want/need them). Yes, this adds more POMs, but downstream it'll be better
              3. I'd like a lot of the cleanup into 2.3
              Show
              Steve Loughran added a comment - Stack, HBase's needs are one of the things I'm thinking of, and once you get into the goal of including Hadoop, HBase and accumulo into a project, things get worse to the extent I couldn't even pull in htmlunit: https://github.com/hortonworks/hoya/blob/master/pom.xml We should really add ZK and avro into the mix here too I'd love to see stripped down clients -which we could address by having some lean poms that only include the core bits to talk to localfs, hdfs, yarn, MR, all optional client-side bits (avro integration, ftpfs, s3fs, pulled to the side where you can add them if you explicitly want/need them). Yes, this adds more POMs, but downstream it'll be better I'd like a lot of the cleanup into 2.3
              Hide
              Permalink
              Colin Patrick McCabe added a comment -

              Thanks for taking a look at this task. It's a difficult one.

              One thing I'd like to add is that we now have httpcore-4.2.5 in trunk combined with httpclient-3.1. However, these two jars provide many of the same classes! The reason is because the Apache httpclient library was end-of-lifed and folded into the httpcore project.

              I am concerned about this dependency since it seems like we should be including one or the other, but not both (given that they provide some of the same classes) I tried getting rid of httpclient 3.1, but it is not possible since we use the custom URL class which is implemented there, and which was dropped in httpcore (they advise using java.net.URI instead). Perhaps we could weed out these uses of the custom URI and try dropping the old client?

              Show
              Colin Patrick McCabe added a comment - Thanks for taking a look at this task. It's a difficult one. One thing I'd like to add is that we now have httpcore-4.2.5 in trunk combined with httpclient-3.1. However, these two jars provide many of the same classes! The reason is because the Apache httpclient library was end-of-lifed and folded into the httpcore project. I am concerned about this dependency since it seems like we should be including one or the other, but not both (given that they provide some of the same classes) I tried getting rid of httpclient 3.1, but it is not possible since we use the custom URL class which is implemented there, and which was dropped in httpcore (they advise using java.net.URI instead). Perhaps we could weed out these uses of the custom URI and try dropping the old client?
              Hide
              Permalink
              Steve Loughran added a comment -

              Colin -that should be a separate JIRA

              I'm trying to triage changes

              1. Low risk, no code changes
              2. minor code changes and/or medium risk
              3. major reworks and/or dependencies known to be brittle

              the httpclient stuff is odd as it's actually been pretty reliable -and far better than the java.net code. It's just that there are now two versions in there, which at least don't conflict. What risks/harms is there from leaving it in, other than binary bloat & getting into the classpath of downstream things -which as there aren't any later versions to conflict with, shouldn't be more than an inconvenience downstream.

              Show
              Steve Loughran added a comment - Colin -that should be a separate JIRA I'm trying to triage changes Low risk, no code changes minor code changes and/or medium risk major reworks and/or dependencies known to be brittle the httpclient stuff is odd as it's actually been pretty reliable -and far better than the java.net code. It's just that there are now two versions in there, which at least don't conflict. What risks/harms is there from leaving it in, other than binary bloat & getting into the classpath of downstream things -which as there aren't any later versions to conflict with, shouldn't be more than an inconvenience downstream.
              Hide
              Permalink
              Vinayakumar B added a comment -

              Thanks Steve for initiating this major task.
              Adding to above points, there are many duplicate jars in the distribution.
              such as,
              hdfs/lib, mapreduce/lib, tools/lib and yarn/lib contains ~90% of the same jars present in common/lib.

              Is there any specific reason to keep these multiple copies of the jars.?

              Show
              Vinayakumar B added a comment - Thanks Steve for initiating this major task. Adding to above points, there are many duplicate jars in the distribution. such as, hdfs/lib, mapreduce/lib, tools/lib and yarn/lib contains ~90% of the same jars present in common/lib. Is there any specific reason to keep these multiple copies of the jars.?
              Hide
              Permalink
              Steve Loughran added a comment -

              Vinay -I think that's a separate issue, as its packaging in the tar files. Feel free to open a JIRA on it, but I'm only looking at the POM files and their implications for downstream projects

              Show
              Steve Loughran added a comment - Vinay -I think that's a separate issue, as its packaging in the tar files. Feel free to open a JIRA on it, but I'm only looking at the POM files and their implications for downstream projects
              Hide
              Permalink
              Vinayakumar B added a comment -

              Ok Steve. No problem.
              Filed HADOOP-10115 for the packaging duplicate jars issue. Thanks

              Show
              Vinayakumar B added a comment - Ok Steve. No problem. Filed HADOOP-10115 for the packaging duplicate jars issue. Thanks
              Hide
              Permalink
              Steve Loughran added a comment -

              HADOOP-10147 covers the commons-logging update

              Show
              Steve Loughran added a comment - HADOOP-10147 covers the commons-logging update
              Hide
              Permalink
              Steve Loughran added a comment -

              Link to HADOOP-9961 which pushed up a couple of dependencies

              Show
              Steve Loughran added a comment - Link to HADOOP-9961 which pushed up a couple of dependencies
              Hide
              Permalink
              Steve Loughran added a comment -

              HADOOP-9555 updates ZK to 3.4.6

              Show
              Steve Loughran added a comment - HADOOP-9555 updates ZK to 3.4.6
              Hide
              Permalink
              Steve Loughran added a comment -

              HDFS-7376 proposes upgrading jsch to avoid java7 problems

              Show
              Steve Loughran added a comment - HDFS-7376 proposes upgrading jsch to avoid java7 problems

                People

                • Assignee:
                  Steve Loughran
                  Reporter:
                  Steve Loughran
                • Votes:
                  2 Vote for this issue
                  Watchers:
                  22 Start watching this issue

                  Dates

                  • Created:
                    Updated:

                    Development