Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9991

Fix up Hadoop POMs, roll up JARs to latest versions

        Details

        • Type: Improvement
        • Status: Open
        • Priority: Major
        • Resolution: Unresolved
        • Affects Version/s: 2.1.1-beta, 2.3.0
        • Fix Version/s: None
        • Component/s: build
        • Labels:
          None
        • Target Version/s:

          Description

          If you try using Hadoop downstream with a classpath shared with HBase and Accumulo, you soon discover how messy the dependencies are.

          Hadoop's side of this problem is

          1. not being up to date with some of the external releases of common JARs
          2. not locking down/excluding inconsistent versions of artifacts provided down the dependency graph

            Issue Links

            depends upon

            Sub-task - The sub-task of the issue HADOOP-14490 Upgrade azure-storage sdk version >5.2.0

            • Major - Major loss of function.
            • Open

            Sub-task - The sub-task of the issue HADOOP-13050 Upgrade to AWS SDK 1.11.45

            • Blocker - Blocks development and/or testing work, production could not run
            • Resolved

            Sub-task - The sub-task of the issue HADOOP-9613 [JDK8] Update jersey version to latest 1.x release

            • Major - Major loss of function.
            • Resolved

            Sub-task - The sub-task of the issue HADOOP-14040 Use shaded aws-sdk uber-JAR 1.11.86

            • Major - Major loss of function.
            • Resolved

            Sub-task - The sub-task of the issue HADOOP-13541 explicitly declare the Joda time version S3A depends on

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Resolved

            Sub-task - The sub-task of the issue HDFS-5411 Update Bookkeeper dependency to 4.2.3

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Closed

            Sub-task - The sub-task of the issue HADOOP-14126 remove jackson, joda and other transient aws SDK dependencies from hadoop-aws

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Patch Available

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-14100 Upgrade Jsch jar to latest version to fix vulnerability in old versions

            • Critical - Crashes, loss of data, severe memory leak.
            • Resolved

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-14283 Upgrade AWS SDK to 1.11.134

            • Critical - Crashes, loss of data, severe memory leak.
            • Resolved

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-11515 Upgrade jsch lib to jsch-0.1.51 to avoid problems running on java7

            • Major - Major loss of function.
            • Resolved

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-12767 update apache httpclient version to 4.5.2; httpcore to 4.4.4

            • Major - Major loss of function.
            • Resolved

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-13994 explicitly declare the commons-lang3 dependency as 3.4

            • Major - Major loss of function.
            • Resolved

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-11086 Upgrade jets3t to 0.9.4

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Resolved

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-11946 Update jets3t jar to 0.9.4 in order to fix leading '/' problem with S3 calls.

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Resolved

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-13545 Upgrade HSQLDB to 2.3.4

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Resolved

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-12577 Bump up commons-collections version to 3.2.2 to address a security flaw

            • Blocker - Blocks development and/or testing work, production could not run
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-9555 HA functionality that uses ZooKeeper may experience inadvertent TCP RST and miss session expiration event due to bug in client connection management

            • Major - Major loss of function.
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-10814 Update Tomcat version used by HttpFS and KMS to latest 6.x version

            • Major - Major loss of function.
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-5431 Missing pom dependency in MR-client

            • Major - Major loss of function.
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-10147 Upgrade to commons-logging 1.1.3 to avoid potential deadlock in MiniDFSCluster

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-14420 generateReports property is not applicable for maven-site-plugin:attach-descriptor goal

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Patch Available

            New Feature - A new feature of the product, which has yet to be developed. HADOOP-12281 Add an incompatible version of Kryo to the hadoop-client classpath

            • Major - Major loss of function.
            • Resolved

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13363 Upgrade protobuf from 2.5.0 to something newer

            • Blocker - Blocks development and/or testing work, production could not run
            • Open

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-10076 Update tomcat/jasper dependency to version 7

            • Major - Major loss of function.
            • Open

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-12928 Update netty to 3.10.5.Final to sync with zookeeper

            • Major - Major loss of function.
            • Open

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-12527 Upgrade Avro dependency to 1.7.7 or later

            • Major - Major loss of function.
            • In Progress

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-12064 [JDK8] Update guice version to 4.0

            • Blocker - Blocks development and/or testing work, production could not run
            • Resolved

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-10075 Update jetty dependency to version 9

            • Critical - Crashes, loss of data, severe memory leak.
            • Resolved

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-10101 Update guava dependency to the latest version

            • Major - Major loss of function.
            • Resolved

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-12927 Update netty-all to 4.0.34.Final

            • Major - Major loss of function.
            • Resolved

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-14401 maven-project-info-reports-plugin can be removed

            • Major - Major loss of function.
            • Resolved

            Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-5624 move grizzly-test and junit dependencies to test scope

            • Major - Major loss of function.
            • Resolved

            Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-5678 Move junit to test scope in more projects

            • Major - Major loss of function.
            • Resolved

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-9244 Upgrade servlet-api dependency from version 2.5 to 3.0.

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Resolved

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-10384 CLONE - Upgrade servlet-api dependency from version 2.5 to 3.0.

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Resolved

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-14419 Remove findbugs report from docs profile

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Resolved

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-9594 Update apache commons math dependency

            • Major - Major loss of function.
            • Closed

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-9611 mvn-rpmbuild against google-guice > 3.0 yields missing cglib dependency

            • Major - Major loss of function.
            • Closed

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-9833 move slf4j to version 1.7.5

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Closed

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13866 Upgrade netty-all to 4.1.1.Final

            • Major - Major loss of function.
            • Patch Available

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-14471 Upgrade Jetty to latest 9.3 version

            • Major - Major loss of function.
            • Patch Available

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-12427 [JDK8] Upgrade Mockito version to 1.10.19

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Patch Available

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-14414 Calling maven-site-plugin directly for docs profile is unnecessary

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Patch Available

            Improvement - An improvement or enhancement to an existing feature or task. YARN-3774 ZKRMStateStore should use CuratorOp when we upgrade to Curator 3

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Patch Available
            incorporates

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13659 Upgrade jaxb-api version

            • Major - Major loss of function.
            • Resolved

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13660 Upgrade commons-configuration version to 2.1

            • Major - Major loss of function.
            • Resolved

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13661 Upgrade HTrace version

            • Major - Major loss of function.
            • Resolved
            is blocked by

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-11755 Update avro version to have PowerPC supported Snappy-java

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Open
            is related to

            Sub-task - The sub-task of the issue HADOOP-14290 Update SLF4J from 1.7.10 to 1.7.25

            • Major - Major loss of function.
            • Resolved

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-10783 apache-commons-lang.jar 2.6 does not support FreeBSD -upgrade to 3.x needed

            • Major - Major loss of function.
            • Open

            Bug - A problem which impairs or prevents the functions of the product. BOOKKEEPER-708 Shade protobuf library to avoid incompatible versions

            • Major - Major loss of function.
            • Resolved

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-9905 remove dependency of zookeeper for hadoop-client

            • Major - Major loss of function.
            • Resolved

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-8793 hadoop-core has dependencies on two different versions of commons-httpclient

            • Critical - Crashes, loss of data, severe memory leak.
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-10100 MiniKDC shouldn't use apacheds-all artifact

            • Major - Major loss of function.
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-9961 versions of a few transitive dependencies diverged between hadoop subprojects

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Closed

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-14043 Shade netty 4 dependency in hadoop-hdfs

            • Critical - Crashes, loss of data, severe memory leak.
            • Open

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-10105 remove httpclient dependency

            • Blocker - Blocks development and/or testing work, production could not run
            • Resolved

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-14061 Update checkstyle version to 6.16 or upper

            • Major - Major loss of function.
            • Resolved

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-10530 Make hadoop trunk build on Java7+ only

            • Blocker - Blocks development and/or testing work, production could not run
            • Closed

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-12232 Upgrade Tomcat dependency to 6.0.44.

            • Major - Major loss of function.
            • Closed

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-10067 Missing POM dependency on jsr305

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Closed

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-12809 Move to tomcat 8.0.21 to support different passwords for key and keystore on HTTPFS/KMS using SSL

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Patch Available
            relates to

            Task - A task that needs to be done. HADOOP-11492 Bump up curator version to 2.7.1

            • Major - Major loss of function.
            • Closed
            requires

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-13332 Remove jackson 1.9.13 and switch all jackson code to 2.x code line

            • Major - Major loss of function.
            • Open

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              stevel@apache.org Steve Loughran added a comment -

              Proposed Fixes

              1. turn on maven enforcement in the build so highlight inconsistencies coming in from below (e.g Avro's SLF4J dependency)
              2. fix those inconsistencies by excluding the conflicts coming from dependencies
              3. Add explicit imports and scope limits on all dependencies, with version numbers we manage
              4. Tightening the downstream exported dependencies, so hadoop-client only declares dependencies on the JARs it really needs (not, say JUnit).
              5. enum later versions of JARs that we can easily migrate to simply by incrementing JARs: and doing that in trunk, with the enforcer identifying more dependency problems to address.
              6. identifying low-cost updates (ideally those with patches in, like the JetS3t/S3 patch), and selectively applying them -again, fixing problems.

              I'd push this all at trunk, though items 1-4 could be backported to 2.x once complete

              Show
              stevel@apache.org Steve Loughran added a comment - Proposed Fixes turn on maven enforcement in the build so highlight inconsistencies coming in from below (e.g Avro's SLF4J dependency) fix those inconsistencies by excluding the conflicts coming from dependencies Add explicit imports and scope limits on all dependencies, with version numbers we manage Tightening the downstream exported dependencies, so hadoop-client only declares dependencies on the JARs it really needs (not, say JUnit). enum later versions of JARs that we can easily migrate to simply by incrementing JARs: and doing that in trunk, with the enforcer identifying more dependency problems to address. identifying low-cost updates (ideally those with patches in, like the JetS3t/S3 patch), and selectively applying them -again, fixing problems. I'd push this all at trunk, though items 1-4 could be backported to 2.x once complete
              Hide
              Permalink
              stevel@apache.org Steve Loughran added a comment -

              enforcement is on, it's more excessive export to things downstream, especially with hbase in the mix -as that is where version number problems start to surface

              This is the main set of "exclusions because they don't appear used" values, though HDFS's JSP pages may well need jasper. The jersey-test-framework-grizzly2 dependency (branch-2.1.1) is clearly spurious

                  <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-minicluster</artifactId>
      <version>${hadoop.version}</version>
      <scope>test</scope>
      <exclusions>
        <exclusion>
          <groupId>com.sun.jersey.jersey-test-framework</groupId>
          <artifactId>jersey-test-framework-grizzly2</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-hdfs</artifactId>
      <version>${hadoop.version}</version>
      <exclusions>
        <exclusion>
          <groupId>tomcat</groupId>
          <artifactId>jasper-runtime</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-yarn-server-common</artifactId>
      <version>${hadoop.version}</version>
      <exclusions>
        <exclusion>
          <groupId>com.sun.jersey.jersey-test-framework</groupId>
          <artifactId>jersey-test-framework-grizzly2</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-yarn-client</artifactId>
      <version>${hadoop.version}</version>
      <exclusions>
        <exclusion>
          <groupId>com.sun.jersey.jersey-test-framework</groupId>
          <artifactId>jersey-test-framework-grizzly2</artifactId>
        </exclusion>
      </exclusions>      
    </dependency>
              Show
              stevel@apache.org Steve Loughran added a comment - enforcement is on, it's more excessive export to things downstream, especially with hbase in the mix -as that is where version number problems start to surface This is the main set of "exclusions because they don't appear used" values, though HDFS's JSP pages may well need jasper. The jersey-test-framework-grizzly2 dependency (branch-2.1.1) is clearly spurious <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minicluster</artifactId> <version>${hadoop.version}</version> <scope>test</scope> <exclusions> <exclusion> <groupId>com.sun.jersey.jersey-test-framework</groupId> <artifactId>jersey-test-framework-grizzly2</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>${hadoop.version}</version> <exclusions> <exclusion> <groupId>tomcat</groupId> <artifactId>jasper-runtime</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-yarn-server-common</artifactId> <version>${hadoop.version}</version> <exclusions> <exclusion> <groupId>com.sun.jersey.jersey-test-framework</groupId> <artifactId>jersey-test-framework-grizzly2</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-yarn-client</artifactId> <version>${hadoop.version}</version> <exclusions> <exclusion> <groupId>com.sun.jersey.jersey-test-framework</groupId> <artifactId>jersey-test-framework-grizzly2</artifactId> </exclusion> </exclusions> </dependency>
              Hide
              Permalink
              stevel@apache.org Steve Loughran added a comment -

              MapReduce is marking junit as compile scoped, not test, so trickles into the tarballs and downstream

              Show
              stevel@apache.org Steve Loughran added a comment - MapReduce is marking junit as compile scoped, not test, so trickles into the tarballs and downstream
              Hide
              Permalink
              stevel@apache.org Steve Loughran added a comment -

              downstream of the hadoop-client artifact, this is what you get on your dependency graph

              [INFO] \- org.apache.hadoop:hadoop-client:pom:2.1.2-SNAPSHOT:compile
[INFO]    +- org.apache.hadoop:hadoop-common:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  +- commons-cli:commons-cli:jar:1.2:compile
[INFO]    |  +- org.apache.commons:commons-math:jar:2.1:compile
[INFO]    |  +- xmlenc:xmlenc:jar:0.52:compile
[INFO]    |  +- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO]    |  +- commons-io:commons-io:jar:2.1:compile
[INFO]    |  +- commons-logging:commons-logging:jar:1.1.1:compile
[INFO]    |  +- log4j:log4j:jar:1.2.17:compile
[INFO]    |  +- commons-lang:commons-lang:jar:2.5:compile
[INFO]    |  +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO]    |  |  +- commons-collections:commons-collections:jar:3.2.1:compile
[INFO]    |  |  +- commons-digester:commons-digester:jar:1.8:compile
[INFO]    |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO]    |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO]    |  +- org.slf4j:slf4j-log4j12:jar:1.7.5:compile
[INFO]    |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.7:compile
[INFO]    |  +- org.apache.avro:avro:jar:1.7.4:compile
[INFO]    |  |  +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO]    |  |  \- org.xerial.snappy:snappy-java:jar:1.0.4.1:compile
[INFO]    |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO]    |  +- org.apache.hadoop:hadoop-auth:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
[INFO]    |  \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO]    |     \- org.tukaani:xz:jar:1.0:compile
[INFO]    +- org.apache.hadoop:hadoop-hdfs:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  +- com.google.guava:guava:jar:11.0.2:compile
[INFO]    |  |  \- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO]    |  +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO]    |  +- commons-codec:commons-codec:jar:1.4:compile
[INFO]    |  \- org.codehaus.jackson:jackson-core-asl:jar:1.9.7:compile
[INFO]    +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  |  +- org.apache.hadoop:hadoop-yarn-client:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  |  |  +- com.google.inject:guice:jar:3.0:compile
[INFO]    |  |  |  |  +- javax.inject:javax.inject:jar:1:compile
[INFO]    |  |  |  |  \- aopalliance:aopalliance:jar:1.0:compile
[INFO]    |  |  |  +- com.sun.jersey:jersey-server:jar:1.9:compile
[INFO]    |  |  |  |  +- asm:asm:jar:3.1:compile
[INFO]    |  |  |  |  \- com.sun.jersey:jersey-core:jar:1.9:compile
[INFO]    |  |  |  +- com.sun.jersey:jersey-json:jar:1.9:compile
[INFO]    |  |  |  |  +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO]    |  |  |  |  |  \- stax:stax-api:jar:1.0.1:compile
[INFO]    |  |  |  |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO]    |  |  |  |  |  \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
[INFO]    |  |  |  |  |     \- javax.activation:activation:jar:1.1:compile
[INFO]    |  |  |  |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile
[INFO]    |  |  |  |  \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile
[INFO]    |  |  |  \- com.sun.jersey.contribs:jersey-guice:jar:1.9:compile
[INFO]    |  |  \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  +- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  \- org.slf4j:slf4j-api:jar:1.7.5:compile
[INFO]    +- org.apache.hadoop:hadoop-yarn-api:jar:2.1.2-SNAPSHOT:compile
[INFO]    +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.1.2-SNAPSHOT:compile
[INFO]    +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.1.2-SNAPSHOT:compile
[INFO]    \- org.apache.hadoop:hadoop-annotations:jar:2.1.2-SNAPSHOT:compile
              Show
              stevel@apache.org Steve Loughran added a comment - downstream of the hadoop-client artifact, this is what you get on your dependency graph [INFO] \- org.apache.hadoop:hadoop-client:pom:2.1.2-SNAPSHOT:compile [INFO] +- org.apache.hadoop:hadoop-common:jar:2.1.2-SNAPSHOT:compile [INFO] | +- commons-cli:commons-cli:jar:1.2:compile [INFO] | +- org.apache.commons:commons-math:jar:2.1:compile [INFO] | +- xmlenc:xmlenc:jar:0.52:compile [INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:compile [INFO] | +- commons-io:commons-io:jar:2.1:compile [INFO] | +- commons-logging:commons-logging:jar:1.1.1:compile [INFO] | +- log4j:log4j:jar:1.2.17:compile [INFO] | +- commons-lang:commons-lang:jar:2.5:compile [INFO] | +- commons-configuration:commons-configuration:jar:1.6:compile [INFO] | | +- commons-collections:commons-collections:jar:3.2.1:compile [INFO] | | +- commons-digester:commons-digester:jar:1.8:compile [INFO] | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile [INFO] | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile [INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.5:compile [INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.7:compile [INFO] | +- org.apache.avro:avro:jar:1.7.4:compile [INFO] | | +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile [INFO] | | \- org.xerial.snappy:snappy-java:jar:1.0.4.1:compile [INFO] | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile [INFO] | +- org.apache.hadoop:hadoop-auth:jar:2.1.2-SNAPSHOT:compile [INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile [INFO] | \- org.apache.commons:commons-compress:jar:1.4.1:compile [INFO] | \- org.tukaani:xz:jar:1.0:compile [INFO] +- org.apache.hadoop:hadoop-hdfs:jar:2.1.2-SNAPSHOT:compile [INFO] | +- com.google.guava:guava:jar:11.0.2:compile [INFO] | | \- com.google.code.findbugs:jsr305:jar:1.3.9:compile [INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile [INFO] | +- commons-codec:commons-codec:jar:1.4:compile [INFO] | \- org.codehaus.jackson:jackson-core-asl:jar:1.9.7:compile [INFO] +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.1.2-SNAPSHOT:compile [INFO] | +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.1.2-SNAPSHOT:compile [INFO] | | +- org.apache.hadoop:hadoop-yarn-client:jar:2.1.2-SNAPSHOT:compile [INFO] | | | +- com.google.inject:guice:jar:3.0:compile [INFO] | | | | +- javax.inject:javax.inject:jar:1:compile [INFO] | | | | \- aopalliance:aopalliance:jar:1.0:compile [INFO] | | | +- com.sun.jersey:jersey-server:jar:1.9:compile [INFO] | | | | +- asm:asm:jar:3.1:compile [INFO] | | | | \- com.sun.jersey:jersey-core:jar:1.9:compile [INFO] | | | +- com.sun.jersey:jersey-json:jar:1.9:compile [INFO] | | | | +- org.codehaus.jettison:jettison:jar:1.1:compile [INFO] | | | | | \- stax:stax-api:jar:1.0.1:compile [INFO] | | | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile [INFO] | | | | | \- javax.xml.bind:jaxb-api:jar:2.2.2:compile [INFO] | | | | | \- javax.activation:activation:jar:1.1:compile [INFO] | | | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile [INFO] | | | | \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile [INFO] | | | \- com.sun.jersey.contribs:jersey-guice:jar:1.9:compile [INFO] | | \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.1.2-SNAPSHOT:compile [INFO] | +- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.1.2-SNAPSHOT:compile [INFO] | \- org.slf4j:slf4j-api:jar:1.7.5:compile [INFO] +- org.apache.hadoop:hadoop-yarn-api:jar:2.1.2-SNAPSHOT:compile [INFO] +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.1.2-SNAPSHOT:compile [INFO] | \- org.apache.hadoop:hadoop-yarn-common:jar:2.1.2-SNAPSHOT:compile [INFO] +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.1.2-SNAPSHOT:compile [INFO] \- org.apache.hadoop:hadoop-annotations:jar:2.1.2-SNAPSHOT:compile
              Hide
              Permalink
              stevel@apache.org Steve Loughran added a comment -
              1. hadoop-hdfs should be excluding com.google.code.findbugs:jsr305:jar:1.3.9 from guava (or downgrade to provided)
              2. hadoop-mapreduce-client-common should not need to depend on yarn-server-common, or jersey-server, though jax
              3. hadoop-common should mark ZK dependency as <provided>, with HDFS server declaring it for HA.
              Show
              stevel@apache.org Steve Loughran added a comment - hadoop-hdfs should be excluding com.google.code.findbugs:jsr305:jar:1.3.9 from guava (or downgrade to provided) hadoop-mapreduce-client-common should not need to depend on yarn-server-common, or jersey-server, though jax hadoop-common should mark ZK dependency as <provided>, with HDFS server declaring it for HA.
              Hide
              Permalink
              stack stack added a comment -

              Steve Loughran Thanks for taking on this masochistic task. Given hadoop surface area, it is hard to figure if a dependency is needed. How about we go radical in trunk since it will be ten years before there is a hadoop3 which should be time enough to figure those dependencies removed that should have been left in. hadoop2 could do w/ a just a purge of at least the just-not-used. I'd like to help out. In hbase we put on a few filters to try and block the plain farcical but god-bless-maven, you have to reproduce this set each time for each hadoop version we build against since no xinclude...

              Show
              stack stack added a comment - Steve Loughran Thanks for taking on this masochistic task. Given hadoop surface area, it is hard to figure if a dependency is needed. How about we go radical in trunk since it will be ten years before there is a hadoop3 which should be time enough to figure those dependencies removed that should have been left in. hadoop2 could do w/ a just a purge of at least the just-not-used. I'd like to help out. In hbase we put on a few filters to try and block the plain farcical but god-bless-maven, you have to reproduce this set each time for each hadoop version we build against since no xinclude...
              Hide
              Permalink
              stevel@apache.org Steve Loughran added a comment -

              Stack, HBase's needs are one of the things I'm thinking of, and once you get into the goal of including Hadoop, HBase and accumulo into a project, things get worse to the extent I couldn't even pull in htmlunit: https://github.com/hortonworks/hoya/blob/master/pom.xml

              1. We should really add ZK and avro into the mix here too
              2. I'd love to see stripped down clients -which we could address by having some lean poms that only include the core bits to talk to localfs, hdfs, yarn, MR, all optional client-side bits (avro integration, ftpfs, s3fs, pulled to the side where you can add them if you explicitly want/need them). Yes, this adds more POMs, but downstream it'll be better
              3. I'd like a lot of the cleanup into 2.3
              Show
              stevel@apache.org Steve Loughran added a comment - Stack, HBase's needs are one of the things I'm thinking of, and once you get into the goal of including Hadoop, HBase and accumulo into a project, things get worse to the extent I couldn't even pull in htmlunit: https://github.com/hortonworks/hoya/blob/master/pom.xml We should really add ZK and avro into the mix here too I'd love to see stripped down clients -which we could address by having some lean poms that only include the core bits to talk to localfs, hdfs, yarn, MR, all optional client-side bits (avro integration, ftpfs, s3fs, pulled to the side where you can add them if you explicitly want/need them). Yes, this adds more POMs, but downstream it'll be better I'd like a lot of the cleanup into 2.3
              Hide
              Permalink
              cmccabe Colin P. McCabe added a comment -

              Thanks for taking a look at this task. It's a difficult one.

              One thing I'd like to add is that we now have httpcore-4.2.5 in trunk combined with httpclient-3.1. However, these two jars provide many of the same classes! The reason is because the Apache httpclient library was end-of-lifed and folded into the httpcore project.

              I am concerned about this dependency since it seems like we should be including one or the other, but not both (given that they provide some of the same classes) I tried getting rid of httpclient 3.1, but it is not possible since we use the custom URL class which is implemented there, and which was dropped in httpcore (they advise using java.net.URI instead). Perhaps we could weed out these uses of the custom URI and try dropping the old client?

              Show
              cmccabe Colin P. McCabe added a comment - Thanks for taking a look at this task. It's a difficult one. One thing I'd like to add is that we now have httpcore-4.2.5 in trunk combined with httpclient-3.1. However, these two jars provide many of the same classes! The reason is because the Apache httpclient library was end-of-lifed and folded into the httpcore project. I am concerned about this dependency since it seems like we should be including one or the other, but not both (given that they provide some of the same classes) I tried getting rid of httpclient 3.1, but it is not possible since we use the custom URL class which is implemented there, and which was dropped in httpcore (they advise using java.net.URI instead). Perhaps we could weed out these uses of the custom URI and try dropping the old client?
              Hide
              Permalink
              stevel@apache.org Steve Loughran added a comment -

              Colin -that should be a separate JIRA

              I'm trying to triage changes

              1. Low risk, no code changes
              2. minor code changes and/or medium risk
              3. major reworks and/or dependencies known to be brittle

              the httpclient stuff is odd as it's actually been pretty reliable -and far better than the java.net code. It's just that there are now two versions in there, which at least don't conflict. What risks/harms is there from leaving it in, other than binary bloat & getting into the classpath of downstream things -which as there aren't any later versions to conflict with, shouldn't be more than an inconvenience downstream.

              Show
              stevel@apache.org Steve Loughran added a comment - Colin -that should be a separate JIRA I'm trying to triage changes Low risk, no code changes minor code changes and/or medium risk major reworks and/or dependencies known to be brittle the httpclient stuff is odd as it's actually been pretty reliable -and far better than the java.net code. It's just that there are now two versions in there, which at least don't conflict. What risks/harms is there from leaving it in, other than binary bloat & getting into the classpath of downstream things -which as there aren't any later versions to conflict with, shouldn't be more than an inconvenience downstream.
              Hide
              Permalink
              vinayrpet Vinayakumar B added a comment -

              Thanks Steve for initiating this major task.
              Adding to above points, there are many duplicate jars in the distribution.
              such as,
              hdfs/lib, mapreduce/lib, tools/lib and yarn/lib contains ~90% of the same jars present in common/lib.

              Is there any specific reason to keep these multiple copies of the jars.?

              Show
              vinayrpet Vinayakumar B added a comment - Thanks Steve for initiating this major task. Adding to above points, there are many duplicate jars in the distribution. such as, hdfs/lib, mapreduce/lib, tools/lib and yarn/lib contains ~90% of the same jars present in common/lib. Is there any specific reason to keep these multiple copies of the jars.?
              Hide
              Permalink
              stevel@apache.org Steve Loughran added a comment -

              Vinay -I think that's a separate issue, as its packaging in the tar files. Feel free to open a JIRA on it, but I'm only looking at the POM files and their implications for downstream projects

              Show
              stevel@apache.org Steve Loughran added a comment - Vinay -I think that's a separate issue, as its packaging in the tar files. Feel free to open a JIRA on it, but I'm only looking at the POM files and their implications for downstream projects
              Hide
              Permalink
              vinayrpet Vinayakumar B added a comment -

              Ok Steve. No problem.
              Filed HADOOP-10115 for the packaging duplicate jars issue. Thanks

              Show
              vinayrpet Vinayakumar B added a comment - Ok Steve. No problem. Filed HADOOP-10115 for the packaging duplicate jars issue. Thanks
              Hide
              Permalink
              stevel@apache.org Steve Loughran added a comment -

              HADOOP-10147 covers the commons-logging update

              Show
              stevel@apache.org Steve Loughran added a comment - HADOOP-10147 covers the commons-logging update
              Hide
              Permalink
              stevel@apache.org Steve Loughran added a comment -

              Link to HADOOP-9961 which pushed up a couple of dependencies

              Show
              stevel@apache.org Steve Loughran added a comment - Link to HADOOP-9961 which pushed up a couple of dependencies
              Hide
              Permalink
              stevel@apache.org Steve Loughran added a comment -

              HADOOP-9555 updates ZK to 3.4.6

              Show
              stevel@apache.org Steve Loughran added a comment - HADOOP-9555 updates ZK to 3.4.6
              Hide
              Permalink
              stevel@apache.org Steve Loughran added a comment -

              HDFS-7376 proposes upgrading jsch to avoid java7 problems

              Show
              stevel@apache.org Steve Loughran added a comment - HDFS-7376 proposes upgrading jsch to avoid java7 problems
              Hide
              Permalink
              prasanth_j Prasanth Jayachandran added a comment -

              Steve Loughran With HIVE-11304 I am trying to migrate to log4j2.x from log4j1.x. Any plans for the same in hadoop?

              Show
              prasanth_j Prasanth Jayachandran added a comment - Steve Loughran With HIVE-11304 I am trying to migrate to log4j2.x from log4j1.x. Any plans for the same in hadoop?
              Hide
              Permalink
              stevel@apache.org Steve Loughran added a comment -

              Hadoop only explicitly gets at log4j in some tests, so code-wise, excluding those, switching to log4j v 2 is technically a matter of changing the mvn version.

              Except

              • we don't know what happens downstream
              • logging is essential to all services
              • rolling logs of YARN services matters too.

              For those reasons, it's hard to see enthusiasm for a change. Especially as async logging has a price: when things fail the log may not be complete.

              What is possible is to move the hadoop logging to slf4j off commons logging; that's something that we'd like to see in new code, even though migrating old code is TBD. Do that and switching to log4j or even logback can be done by changing the slf4j dependency

              Show
              stevel@apache.org Steve Loughran added a comment - Hadoop only explicitly gets at log4j in some tests, so code-wise, excluding those, switching to log4j v 2 is technically a matter of changing the mvn version. Except we don't know what happens downstream logging is essential to all services rolling logs of YARN services matters too. For those reasons, it's hard to see enthusiasm for a change. Especially as async logging has a price: when things fail the log may not be complete. What is possible is to move the hadoop logging to slf4j off commons logging; that's something that we'd like to see in new code, even though migrating old code is TBD. Do that and switching to log4j or even logback can be done by changing the slf4j dependency
              Hide
              Permalink
              githubbot ASF GitHub Bot added a comment -

              Github user steveloughran commented on the pull request:

              https://github.com/apache/hadoop/pull/84#issuecomment-197274477

              Please can you file a separate JIRA for these two changes, link to HADOOP-9991 and include justification. Version updates are a traumatic issue in Hadoop and done fairly reluctantly.

              FWIW, updating netty-all from beta to final makes sense just from a release perspective; updating the other jetty less so

              Show
              githubbot ASF GitHub Bot added a comment - Github user steveloughran commented on the pull request: https://github.com/apache/hadoop/pull/84#issuecomment-197274477 Please can you file a separate JIRA for these two changes, link to HADOOP-9991 and include justification. Version updates are a traumatic issue in Hadoop and done fairly reluctantly. FWIW, updating netty-all from beta to final makes sense just from a release perspective; updating the other jetty less so
              Hide
              Permalink
              githubbot ASF GitHub Bot added a comment -

              Github user ceefour commented on the pull request:

              https://github.com/apache/hadoop/pull/84#issuecomment-197323519

              @steveloughran This pull request now only updates netty-all (minor version update).

              Show
              githubbot ASF GitHub Bot added a comment - Github user ceefour commented on the pull request: https://github.com/apache/hadoop/pull/84#issuecomment-197323519 @steveloughran This pull request now only updates netty-all (minor version update).
              Hide
              Permalink
              githubbot ASF GitHub Bot added a comment -

              GitHub user ceefour opened a pull request:

              https://github.com/apache/hadoop/pull/85

              HADOOP-9991. Update netty to 3.7.1.Final because hadoop-client 2.7.2

              depends on zookeeper 3.4.6 which depends on netty 3.7.x. Related to
              https://github.com/apache/hadoop/pull/84

              You can merge this pull request into a Git repository by running:

              $ git pull https://github.com/ceefour/hadoop HADOOP-9991-netty371

              Alternatively you can review and apply these changes as the patch at:

              https://github.com/apache/hadoop/pull/85.patch

              To close this pull request, make a commit to your master/trunk branch
              with (at least) the following in the commit message:

              This closes #85

              commit 2a2c248f8516e69c8f8bfe979765f597cb1c8729
              Author: Hendy Irawan <ceefour666@gmail.com>
              Date: 2016-03-16T13:18:56Z

              HADOOP-9991. Update netty to 3.7.1.Final because hadoop-client 2.7.2
              depends on zookeeper 3.4.6 which depends on netty 3.7.x. Related to
              https://github.com/apache/hadoop/pull/84

              Show
              githubbot ASF GitHub Bot added a comment - GitHub user ceefour opened a pull request: https://github.com/apache/hadoop/pull/85 HADOOP-9991 . Update netty to 3.7.1.Final because hadoop-client 2.7.2 depends on zookeeper 3.4.6 which depends on netty 3.7.x. Related to https://github.com/apache/hadoop/pull/84 You can merge this pull request into a Git repository by running: $ git pull https://github.com/ceefour/hadoop HADOOP-9991 -netty371 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/85.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #85 commit 2a2c248f8516e69c8f8bfe979765f597cb1c8729 Author: Hendy Irawan <ceefour666@gmail.com> Date: 2016-03-16T13:18:56Z HADOOP-9991 . Update netty to 3.7.1.Final because hadoop-client 2.7.2 depends on zookeeper 3.4.6 which depends on netty 3.7.x. Related to https://github.com/apache/hadoop/pull/84
              Hide
              Permalink
              githubbot ASF GitHub Bot added a comment -

              Github user steveloughran commented on the pull request:

              https://github.com/apache/hadoop/pull/84#issuecomment-197340633

              sorry, I should have been clearer
              -create a new Hadoop JIRA for this, mark it as a dependency of HADOOP-9991. thanks

              Show
              githubbot ASF GitHub Bot added a comment - Github user steveloughran commented on the pull request: https://github.com/apache/hadoop/pull/84#issuecomment-197340633 sorry, I should have been clearer -create a new Hadoop JIRA for this, mark it as a dependency of HADOOP-9991 . thanks
              Hide
              Permalink
              githubbot ASF GitHub Bot added a comment -

              Github user ceefour commented on the pull request:

              https://github.com/apache/hadoop/pull/84#issuecomment-197358937

              @steveloughran created HADOOP-12927

              Show
              githubbot ASF GitHub Bot added a comment - Github user ceefour commented on the pull request: https://github.com/apache/hadoop/pull/84#issuecomment-197358937 @steveloughran created HADOOP-12927
              Hide
              Permalink
              githubbot ASF GitHub Bot added a comment -

              Github user ceefour closed the pull request at:

              https://github.com/apache/zookeeper/pull/57

              Show
              githubbot ASF GitHub Bot added a comment - Github user ceefour closed the pull request at: https://github.com/apache/zookeeper/pull/57

                People

                • Assignee:
                  stevel@apache.org Steve Loughran
                  Reporter:
                  stevel@apache.org Steve Loughran
                • Votes:
                  5 Vote for this issue
                  Watchers:
                  36 Start watching this issue

                  Dates

                  • Created:
                    Updated:

                    Development