[HADOOP-9991] Fix up Hadoop POMs, roll up JARs to latest versions - ASF JIRA

Details

Type: Improvement
Status: Open
Priority: Major
Resolution: Unresolved
Affects Version/s: 2.1.1-beta, 2.3.0
Fix Version/s: None
Component/s: build
Labels:
None

Target Version/s:

Description

If you try using Hadoop downstream with a classpath shared with HBase and Accumulo, you soon discover how messy the dependencies are.

Hadoop's side of this problem is

not being up to date with some of the external releases of common JARs
not locking down/excluding inconsistent versions of artifacts provided down the dependency graph

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending

Attachments

hadoop-9991-v1.txt

13/Nov/13 18:59

5 kB

Ted Yu

Issue Links

depends upon

HADOOP-13050 Upgrade to AWS SDK 1.11.45

Resolved

HADOOP-9613 [JDK8] Update jersey version to latest 1.x release

Resolved

HADOOP-14040 Use shaded aws-sdk uber-JAR 1.11.86

Resolved

HADOOP-14490 Upgrade azure-storage sdk version >5.4.0

Resolved

HADOOP-13541 explicitly declare the Joda time version S3A depends on

Resolved

HADOOP-14126 remove jackson, joda and other transient aws SDK dependencies from hadoop-aws

Resolved

HDFS-5411 Update Bookkeeper dependency to 4.2.3

Closed

HADOOP-14100 Upgrade Jsch jar to latest version to fix vulnerability in old versions

Resolved

HADOOP-14283 Upgrade AWS SDK to 1.11.134

Resolved

HADOOP-11515 Upgrade jsch lib to jsch-0.1.51 to avoid problems running on java7

Resolved

HADOOP-12767 update apache httpclient version to 4.5.2; httpcore to 4.4.4

Resolved

HADOOP-13994 explicitly declare the commons-lang3 dependency as 3.4

Resolved

HADOOP-11086 Upgrade jets3t to 0.9.4

Resolved

HADOOP-11946 Update jets3t jar to 0.9.4 in order to fix leading '/' problem with S3 calls.

Resolved

HADOOP-13545 Upgrade HSQLDB to 2.3.4

Resolved

HADOOP-14420 generateReports property is not applicable for maven-site-plugin:attach-descriptor goal

Resolved

HADOOP-14692 Upgrade Apache Rat

Resolved

HADOOP-12577 Bump up commons-collections version to 3.2.2 to address a security flaw

Closed

HADOOP-9555 HA functionality that uses ZooKeeper may experience inadvertent TCP RST and miss session expiration event due to bug in client connection management

Closed

HADOOP-10814 Update Tomcat version used by HttpFS and KMS to latest 6.x version

Closed

MAPREDUCE-5431 Missing pom dependency in MR-client

Closed

HADOOP-10147 Upgrade to commons-logging 1.1.3 to avoid potential deadlock in MiniDFSCluster

Closed

HADOOP-12281 Add an incompatible version of Kryo to the hadoop-client classpath

Resolved

HADOOP-13363 Upgrade protobuf from 2.5.0 to something newer

Open

HADOOP-10076 Update tomcat/jasper dependency to version 7

Open

HADOOP-12928 Update netty to 3.10.5.Final to sync with zookeeper

Open

HADOOP-14775 Change junit dependency in parent pom file to junit 5 while maintaining backward compatibility to junit4.

Open

HADOOP-12527 Upgrade Avro dependency to 1.7.7 or later

In Progress

HADOOP-12064 [JDK8] Update guice version to 4.0

Resolved

HADOOP-10075 Update jetty dependency to version 9

Resolved

HADOOP-10101 Update guava dependency to the latest version

Resolved

HADOOP-12927 Update netty-all to 4.0.34.Final

Resolved

HADOOP-14401 maven-project-info-reports-plugin can be removed

Resolved

HADOOP-14471 Upgrade Jetty to latest 9.3 version

Resolved

HADOOP-14536 Update azure-storage sdk to version 5.3.0

Resolved

HADOOP-14662 Update azure-storage sdk to version 5.4.0

Resolved

MAPREDUCE-5624 move grizzly-test and junit dependencies to test scope

Resolved

MAPREDUCE-5678 Move junit to test scope in more projects

Resolved

HADOOP-9244 Upgrade servlet-api dependency from version 2.5 to 3.0.

Resolved

HADOOP-10384 CLONE - Upgrade servlet-api dependency from version 2.5 to 3.0.

Resolved

HADOOP-14419 Remove findbugs report from docs profile

Resolved

HADOOP-9594 Update apache commons math dependency

Closed

HADOOP-9611 mvn-rpmbuild against google-guice > 3.0 yields missing cglib dependency

Closed

HADOOP-9833 move slf4j to version 1.7.5

Closed

HADOOP-13866 Upgrade netty-all to 4.1.1.Final

Patch Available

HADOOP-12427 [JDK8] Upgrade Mockito version to 1.10.19

Patch Available

HADOOP-14414 Calling maven-site-plugin directly for docs profile is unnecessary

Patch Available

YARN-3774 ZKRMStateStore should use CuratorOp when we upgrade to Curator 3

Patch Available

incorporates

HADOOP-13659 Upgrade jaxb-api version

Resolved

HADOOP-13660 Upgrade commons-configuration version to 2.1

Resolved

HADOOP-13661 Upgrade HTrace version

Resolved

is blocked by

HADOOP-11755 Update avro version to have PowerPC supported Snappy-java

Open

is related to

HADOOP-14290 Update SLF4J from 1.7.10 to 1.7.25

Resolved

HADOOP-10783 apache-commons-lang.jar 2.6 does not support FreeBSD -upgrade to 3.x needed

Open

BOOKKEEPER-708 Shade protobuf library to avoid incompatible versions

Resolved

HADOOP-9905 remove dependency of zookeeper for hadoop-client

Resolved

HADOOP-8793 hadoop-core has dependencies on two different versions of commons-httpclient

Closed

HADOOP-10100 MiniKDC shouldn't use apacheds-all artifact

Closed

HADOOP-9961 versions of a few transitive dependencies diverged between hadoop subprojects

Closed

HADOOP-14043 Shade netty 4 dependency in hadoop-hdfs

Open

HADOOP-14670 Increase minimum cmake version for all platforms

Open

HADOOP-10105 remove httpclient dependency

Resolved

HADOOP-14061 Update checkstyle version to 6.16 or upper

Resolved

HADOOP-10530 Make hadoop trunk build on Java7+ only

Closed

HADOOP-12232 Upgrade Tomcat dependency to 6.0.44.

Closed

HADOOP-10067 Missing POM dependency on jsr305

Closed

HADOOP-12809 Move to tomcat 8.0.21 to support different passwords for key and keystore on HTTPFS/KMS using SSL

Patch Available

HADOOP-14647 Update third-party libraries for Hadoop 3

Open

relates to

HADOOP-11492 Bump up curator version to 2.7.1

Closed

requires

HADOOP-13332 Remove jackson 1.9.13 and switch all jackson code to 2.x code line

Open

Show 65 more links (43 depends upon, 3 incorporates, 1 is blocked by, 16 is related to, 1 relates to, 1 requires)

Options
- Show All
- Show Open

Sub-Tasks

1.	update commons IO from 2.1 to 2.4	Closed	Akira Ajisaka
2.	update commons-lang to 2.6	Closed	Akira Ajisaka
3.	Update jackson to 1.9.13	Closed	Akira Ajisaka
4.	Move junit up to v 4.11	Closed	Chris Nauroth
5.	Bump the maven plugin versions too -moving the numbers into properties	Closed	Akira Ajisaka
6.	Increment SLF4J version to 1.7.10	Closed	Tim Robertson
7.	Update Guava to 18.0	Resolved	Unassigned
8.	Update aws-sdk dependency to 1.10.6; move to aws-sdk-s3	Resolved	Thomas Demoor
9.	Upgrade Jackson 2.2.3 to 2.7.8	Resolved	Sean Mackrory
10.	[JDK8] Update jersey version to latest 1.x release	Resolved	Tsuyoshi Ozawa
11.	Update jetty dependencies	Resolved	Timothy St. Clair
12.	clean up POM dependencies	Open	Alejandro Abdelnur
13.	Move to netty 4.1.x release	Resolved	Unassigned
14.	shade protobuf in the hadoop-common jar	Resolved	Unassigned
15.	Upgrade Avro to 1.8.x	Resolved	Unassigned
16.	Move Mockito up to version 2.x	Open	Akira Ajisaka

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Steve Loughran added a comment - 23/Sep/13 08:52

Proposed Fixes

turn on maven enforcement in the build so highlight inconsistencies coming in from below (e.g Avro's SLF4J dependency)
fix those inconsistencies by excluding the conflicts coming from dependencies
Add explicit imports and scope limits on all dependencies, with version numbers we manage
Tightening the downstream exported dependencies, so hadoop-client only declares dependencies on the JARs it really needs (not, say JUnit).
enum later versions of JARs that we can easily migrate to simply by incrementing JARs: and doing that in trunk, with the enforcer identifying more dependency problems to address.
identifying low-cost updates (ideally those with patches in, like the JetS3t/S3 patch), and selectively applying them -again, fixing problems.

I'd push this all at trunk, though items 1-4 could be backported to 2.x once complete

Show

Steve Loughran added a comment - 23/Sep/13 08:52 Proposed Fixes turn on maven enforcement in the build so highlight inconsistencies coming in from below (e.g Avro's SLF4J dependency) fix those inconsistencies by excluding the conflicts coming from dependencies Add explicit imports and scope limits on all dependencies, with version numbers we manage Tightening the downstream exported dependencies, so hadoop-client only declares dependencies on the JARs it really needs (not, say JUnit). enum later versions of JARs that we can easily migrate to simply by incrementing JARs: and doing that in trunk, with the enforcer identifying more dependency problems to address. identifying low-cost updates (ideally those with patches in, like the JetS3t/S3 patch), and selectively applying them -again, fixing problems. I'd push this all at trunk, though items 1-4 could be backported to 2.x once complete

Hide

Permalink

Steve Loughran added a comment - 01/Oct/13 17:27

enforcement is on, it's more excessive export to things downstream, especially with hbase in the mix -as that is where version number problems start to surface

This is the main set of "exclusions because they don't appear used" values, though HDFS's JSP pages may well need jasper. The jersey-test-framework-grizzly2 dependency (branch-2.1.1) is clearly spurious

    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-minicluster</artifactId>
      <version>${hadoop.version}</version>
      <scope>test</scope>
      <exclusions>
        <exclusion>
          <groupId>com.sun.jersey.jersey-test-framework</groupId>
          <artifactId>jersey-test-framework-grizzly2</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-hdfs</artifactId>
      <version>${hadoop.version}</version>
      <exclusions>
        <exclusion>
          <groupId>tomcat</groupId>
          <artifactId>jasper-runtime</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-yarn-server-common</artifactId>
      <version>${hadoop.version}</version>
      <exclusions>
        <exclusion>
          <groupId>com.sun.jersey.jersey-test-framework</groupId>
          <artifactId>jersey-test-framework-grizzly2</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-yarn-client</artifactId>
      <version>${hadoop.version}</version>
      <exclusions>
        <exclusion>
          <groupId>com.sun.jersey.jersey-test-framework</groupId>
          <artifactId>jersey-test-framework-grizzly2</artifactId>
        </exclusion>
      </exclusions>      
    </dependency>

Show

Steve Loughran added a comment - 01/Oct/13 17:27 enforcement is on, it's more excessive export to things downstream, especially with hbase in the mix -as that is where version number problems start to surface This is the main set of "exclusions because they don't appear used" values, though HDFS's JSP pages may well need jasper. The jersey-test-framework-grizzly2 dependency (branch-2.1.1) is clearly spurious <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minicluster</artifactId> <version>${hadoop.version}</version> <scope>test</scope> <exclusions> <exclusion> <groupId>com.sun.jersey.jersey-test-framework</groupId> <artifactId>jersey-test-framework-grizzly2</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>${hadoop.version}</version> <exclusions> <exclusion> <groupId>tomcat</groupId> <artifactId>jasper-runtime</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-yarn-server-common</artifactId> <version>${hadoop.version}</version> <exclusions> <exclusion> <groupId>com.sun.jersey.jersey-test-framework</groupId> <artifactId>jersey-test-framework-grizzly2</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-yarn-client</artifactId> <version>${hadoop.version}</version> <exclusions> <exclusion> <groupId>com.sun.jersey.jersey-test-framework</groupId> <artifactId>jersey-test-framework-grizzly2</artifactId> </exclusion> </exclusions> </dependency>

Hide

Permalink

Steve Loughran added a comment - 02/Oct/13 15:16

MapReduce is marking junit as compile scoped, not test, so trickles into the tarballs and downstream

Show

Steve Loughran added a comment - 02/Oct/13 15:16 MapReduce is marking junit as compile scoped, not test, so trickles into the tarballs and downstream

Hide

Permalink

Steve Loughran added a comment - 03/Oct/13 13:25

downstream of the hadoop-client artifact, this is what you get on your dependency graph

[INFO] \- org.apache.hadoop:hadoop-client:pom:2.1.2-SNAPSHOT:compile
[INFO]    +- org.apache.hadoop:hadoop-common:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  +- commons-cli:commons-cli:jar:1.2:compile
[INFO]    |  +- org.apache.commons:commons-math:jar:2.1:compile
[INFO]    |  +- xmlenc:xmlenc:jar:0.52:compile
[INFO]    |  +- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO]    |  +- commons-io:commons-io:jar:2.1:compile
[INFO]    |  +- commons-logging:commons-logging:jar:1.1.1:compile
[INFO]    |  +- log4j:log4j:jar:1.2.17:compile
[INFO]    |  +- commons-lang:commons-lang:jar:2.5:compile
[INFO]    |  +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO]    |  |  +- commons-collections:commons-collections:jar:3.2.1:compile
[INFO]    |  |  +- commons-digester:commons-digester:jar:1.8:compile
[INFO]    |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO]    |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO]    |  +- org.slf4j:slf4j-log4j12:jar:1.7.5:compile
[INFO]    |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.7:compile
[INFO]    |  +- org.apache.avro:avro:jar:1.7.4:compile
[INFO]    |  |  +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO]    |  |  \- org.xerial.snappy:snappy-java:jar:1.0.4.1:compile
[INFO]    |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO]    |  +- org.apache.hadoop:hadoop-auth:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
[INFO]    |  \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO]    |     \- org.tukaani:xz:jar:1.0:compile
[INFO]    +- org.apache.hadoop:hadoop-hdfs:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  +- com.google.guava:guava:jar:11.0.2:compile
[INFO]    |  |  \- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO]    |  +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO]    |  +- commons-codec:commons-codec:jar:1.4:compile
[INFO]    |  \- org.codehaus.jackson:jackson-core-asl:jar:1.9.7:compile
[INFO]    +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  |  +- org.apache.hadoop:hadoop-yarn-client:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  |  |  +- com.google.inject:guice:jar:3.0:compile
[INFO]    |  |  |  |  +- javax.inject:javax.inject:jar:1:compile
[INFO]    |  |  |  |  \- aopalliance:aopalliance:jar:1.0:compile
[INFO]    |  |  |  +- com.sun.jersey:jersey-server:jar:1.9:compile
[INFO]    |  |  |  |  +- asm:asm:jar:3.1:compile
[INFO]    |  |  |  |  \- com.sun.jersey:jersey-core:jar:1.9:compile
[INFO]    |  |  |  +- com.sun.jersey:jersey-json:jar:1.9:compile
[INFO]    |  |  |  |  +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO]    |  |  |  |  |  \- stax:stax-api:jar:1.0.1:compile
[INFO]    |  |  |  |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO]    |  |  |  |  |  \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
[INFO]    |  |  |  |  |     \- javax.activation:activation:jar:1.1:compile
[INFO]    |  |  |  |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile
[INFO]    |  |  |  |  \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile
[INFO]    |  |  |  \- com.sun.jersey.contribs:jersey-guice:jar:1.9:compile
[INFO]    |  |  \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  +- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  \- org.slf4j:slf4j-api:jar:1.7.5:compile
[INFO]    +- org.apache.hadoop:hadoop-yarn-api:jar:2.1.2-SNAPSHOT:compile
[INFO]    +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.1.2-SNAPSHOT:compile
[INFO]    |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.1.2-SNAPSHOT:compile
[INFO]    +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.1.2-SNAPSHOT:compile
[INFO]    \- org.apache.hadoop:hadoop-annotations:jar:2.1.2-SNAPSHOT:compile

Show

Steve Loughran added a comment - 03/Oct/13 13:25 downstream of the hadoop-client artifact, this is what you get on your dependency graph [INFO] \- org.apache.hadoop:hadoop-client:pom:2.1.2-SNAPSHOT:compile [INFO] +- org.apache.hadoop:hadoop-common:jar:2.1.2-SNAPSHOT:compile [INFO] | +- commons-cli:commons-cli:jar:1.2:compile [INFO] | +- org.apache.commons:commons-math:jar:2.1:compile [INFO] | +- xmlenc:xmlenc:jar:0.52:compile [INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:compile [INFO] | +- commons-io:commons-io:jar:2.1:compile [INFO] | +- commons-logging:commons-logging:jar:1.1.1:compile [INFO] | +- log4j:log4j:jar:1.2.17:compile [INFO] | +- commons-lang:commons-lang:jar:2.5:compile [INFO] | +- commons-configuration:commons-configuration:jar:1.6:compile [INFO] | | +- commons-collections:commons-collections:jar:3.2.1:compile [INFO] | | +- commons-digester:commons-digester:jar:1.8:compile [INFO] | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile [INFO] | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile [INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.5:compile [INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.7:compile [INFO] | +- org.apache.avro:avro:jar:1.7.4:compile [INFO] | | +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile [INFO] | | \- org.xerial.snappy:snappy-java:jar:1.0.4.1:compile [INFO] | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile [INFO] | +- org.apache.hadoop:hadoop-auth:jar:2.1.2-SNAPSHOT:compile [INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile [INFO] | \- org.apache.commons:commons-compress:jar:1.4.1:compile [INFO] | \- org.tukaani:xz:jar:1.0:compile [INFO] +- org.apache.hadoop:hadoop-hdfs:jar:2.1.2-SNAPSHOT:compile [INFO] | +- com.google.guava:guava:jar:11.0.2:compile [INFO] | | \- com.google.code.findbugs:jsr305:jar:1.3.9:compile [INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile [INFO] | +- commons-codec:commons-codec:jar:1.4:compile [INFO] | \- org.codehaus.jackson:jackson-core-asl:jar:1.9.7:compile [INFO] +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.1.2-SNAPSHOT:compile [INFO] | +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.1.2-SNAPSHOT:compile [INFO] | | +- org.apache.hadoop:hadoop-yarn-client:jar:2.1.2-SNAPSHOT:compile [INFO] | | | +- com.google.inject:guice:jar:3.0:compile [INFO] | | | | +- javax.inject:javax.inject:jar:1:compile [INFO] | | | | \- aopalliance:aopalliance:jar:1.0:compile [INFO] | | | +- com.sun.jersey:jersey-server:jar:1.9:compile [INFO] | | | | +- asm:asm:jar:3.1:compile [INFO] | | | | \- com.sun.jersey:jersey-core:jar:1.9:compile [INFO] | | | +- com.sun.jersey:jersey-json:jar:1.9:compile [INFO] | | | | +- org.codehaus.jettison:jettison:jar:1.1:compile [INFO] | | | | | \- stax:stax-api:jar:1.0.1:compile [INFO] | | | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile [INFO] | | | | | \- javax.xml.bind:jaxb-api:jar:2.2.2:compile [INFO] | | | | | \- javax.activation:activation:jar:1.1:compile [INFO] | | | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile [INFO] | | | | \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile [INFO] | | | \- com.sun.jersey.contribs:jersey-guice:jar:1.9:compile [INFO] | | \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.1.2-SNAPSHOT:compile [INFO] | +- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.1.2-SNAPSHOT:compile [INFO] | \- org.slf4j:slf4j-api:jar:1.7.5:compile [INFO] +- org.apache.hadoop:hadoop-yarn-api:jar:2.1.2-SNAPSHOT:compile [INFO] +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.1.2-SNAPSHOT:compile [INFO] | \- org.apache.hadoop:hadoop-yarn-common:jar:2.1.2-SNAPSHOT:compile [INFO] +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.1.2-SNAPSHOT:compile [INFO] \- org.apache.hadoop:hadoop-annotations:jar:2.1.2-SNAPSHOT:compile

Hide

Permalink

Steve Loughran added a comment - 03/Oct/13 13:34

hadoop-hdfs should be excluding com.google.code.findbugs:jsr305:jar:1.3.9 from guava (or downgrade to provided)
hadoop-mapreduce-client-common should not need to depend on yarn-server-common, or jersey-server, though jax
hadoop-common should mark ZK dependency as <provided>, with HDFS server declaring it for HA.

Show

Steve Loughran added a comment - 03/Oct/13 13:34 hadoop-hdfs should be excluding com.google.code.findbugs:jsr305:jar:1.3.9 from guava (or downgrade to provided) hadoop-mapreduce-client-common should not need to depend on yarn-server-common, or jersey-server, though jax hadoop-common should mark ZK dependency as <provided>, with HDFS server declaring it for HA.

Hide

Permalink

stack added a comment - 14/Nov/13 01:03

Steve Loughran Thanks for taking on this masochistic task. Given hadoop surface area, it is hard to figure if a dependency is needed. How about we go radical in trunk since it will be ten years before there is a hadoop3 which should be time enough to figure those dependencies removed that should have been left in. hadoop2 could do w/ a just a purge of at least the just-not-used. I'd like to help out. In hbase we put on a few filters to try and block the plain farcical but god-bless-maven, you have to reproduce this set each time for each hadoop version we build against since no xinclude...

Show

stack added a comment - 14/Nov/13 01:03 Steve Loughran Thanks for taking on this masochistic task. Given hadoop surface area, it is hard to figure if a dependency is needed. How about we go radical in trunk since it will be ten years before there is a hadoop3 which should be time enough to figure those dependencies removed that should have been left in. hadoop2 could do w/ a just a purge of at least the just-not-used. I'd like to help out. In hbase we put on a few filters to try and block the plain farcical but god-bless-maven, you have to reproduce this set each time for each hadoop version we build against since no xinclude...

Hide

Permalink

Steve Loughran added a comment - 14/Nov/13 09:23

Stack, HBase's needs are one of the things I'm thinking of, and once you get into the goal of including Hadoop, HBase and accumulo into a project, things get worse to the extent I couldn't even pull in htmlunit: https://github.com/hortonworks/hoya/blob/master/pom.xml

We should really add ZK and avro into the mix here too
I'd love to see stripped down clients -which we could address by having some lean poms that only include the core bits to talk to localfs, hdfs, yarn, MR, all optional client-side bits (avro integration, ftpfs, s3fs, pulled to the side where you can add them if you explicitly want/need them). Yes, this adds more POMs, but downstream it'll be better
I'd like a lot of the cleanup into 2.3

Show

Steve Loughran added a comment - 14/Nov/13 09:23 Stack, HBase's needs are one of the things I'm thinking of, and once you get into the goal of including Hadoop, HBase and accumulo into a project, things get worse to the extent I couldn't even pull in htmlunit: https://github.com/hortonworks/hoya/blob/master/pom.xml We should really add ZK and avro into the mix here too I'd love to see stripped down clients -which we could address by having some lean poms that only include the core bits to talk to localfs, hdfs, yarn, MR, all optional client-side bits (avro integration, ftpfs, s3fs, pulled to the side where you can add them if you explicitly want/need them). Yes, this adds more POMs, but downstream it'll be better I'd like a lot of the cleanup into 2.3

Hide

Permalink

Colin P. McCabe added a comment - 15/Nov/13 00:33

Thanks for taking a look at this task. It's a difficult one.

One thing I'd like to add is that we now have httpcore-4.2.5 in trunk combined with httpclient-3.1. However, these two jars provide many of the same classes! The reason is because the Apache httpclient library was end-of-lifed and folded into the httpcore project.

I am concerned about this dependency since it seems like we should be including one or the other, but not both (given that they provide some of the same classes) I tried getting rid of httpclient 3.1, but it is not possible since we use the custom URL class which is implemented there, and which was dropped in httpcore (they advise using java.net.URI instead). Perhaps we could weed out these uses of the custom URI and try dropping the old client?

Show

Colin P. McCabe added a comment - 15/Nov/13 00:33 Thanks for taking a look at this task. It's a difficult one. One thing I'd like to add is that we now have httpcore-4.2.5 in trunk combined with httpclient-3.1. However, these two jars provide many of the same classes! The reason is because the Apache httpclient library was end-of-lifed and folded into the httpcore project. I am concerned about this dependency since it seems like we should be including one or the other, but not both (given that they provide some of the same classes) I tried getting rid of httpclient 3.1, but it is not possible since we use the custom URL class which is implemented there, and which was dropped in httpcore (they advise using java.net.URI instead). Perhaps we could weed out these uses of the custom URI and try dropping the old client?

Hide

Permalink

Steve Loughran added a comment - 15/Nov/13 11:14

Colin -that should be a separate JIRA

I'm trying to triage changes

Low risk, no code changes
minor code changes and/or medium risk
major reworks and/or dependencies known to be brittle

the httpclient stuff is odd as it's actually been pretty reliable -and far better than the java.net code. It's just that there are now two versions in there, which at least don't conflict. What risks/harms is there from leaving it in, other than binary bloat & getting into the classpath of downstream things -which as there aren't any later versions to conflict with, shouldn't be more than an inconvenience downstream.

Show

Steve Loughran added a comment - 15/Nov/13 11:14 Colin -that should be a separate JIRA I'm trying to triage changes Low risk, no code changes minor code changes and/or medium risk major reworks and/or dependencies known to be brittle the httpclient stuff is odd as it's actually been pretty reliable -and far better than the java.net code. It's just that there are now two versions in there, which at least don't conflict. What risks/harms is there from leaving it in, other than binary bloat & getting into the classpath of downstream things -which as there aren't any later versions to conflict with, shouldn't be more than an inconvenience downstream.

Hide

Permalink

Vinayakumar B added a comment - 17/Nov/13 15:16

Thanks Steve for initiating this major task.
Adding to above points, there are many duplicate jars in the distribution.
such as,
hdfs/lib, mapreduce/lib, tools/lib and yarn/lib contains ~90% of the same jars present in common/lib.

Is there any specific reason to keep these multiple copies of the jars.?

Show

Vinayakumar B added a comment - 17/Nov/13 15:16 Thanks Steve for initiating this major task. Adding to above points, there are many duplicate jars in the distribution. such as, hdfs/lib, mapreduce/lib, tools/lib and yarn/lib contains ~90% of the same jars present in common/lib. Is there any specific reason to keep these multiple copies of the jars.?

Hide

Permalink

Steve Loughran added a comment - 18/Nov/13 10:04

Vinay -I think that's a separate issue, as its packaging in the tar files. Feel free to open a JIRA on it, but I'm only looking at the POM files and their implications for downstream projects

Show

Steve Loughran added a comment - 18/Nov/13 10:04 Vinay -I think that's a separate issue, as its packaging in the tar files. Feel free to open a JIRA on it, but I'm only looking at the POM files and their implications for downstream projects

Hide

Permalink

Vinayakumar B added a comment - 19/Nov/13 06:31

Ok Steve. No problem.
Filed ~~HADOOP-10115~~ for the packaging duplicate jars issue. Thanks

Show

Vinayakumar B added a comment - 19/Nov/13 06:31 Ok Steve. No problem. Filed HADOOP-10115 for the packaging duplicate jars issue. Thanks

Hide

Permalink

Steve Loughran added a comment - 07/Dec/13 09:19

~~HADOOP-10147~~ covers the commons-logging update

Show

Steve Loughran added a comment - 07/Dec/13 09:19 HADOOP-10147 covers the commons-logging update

Hide

Permalink

Steve Loughran added a comment - 28/Apr/14 20:54

Link to ~~HADOOP-9961~~ which pushed up a couple of dependencies

Show

Steve Loughran added a comment - 28/Apr/14 20:54 Link to HADOOP-9961 which pushed up a couple of dependencies

Hide

Permalink

Steve Loughran added a comment - 17/Jun/14 17:55

~~HADOOP-9555~~ updates ZK to 3.4.6

Show

Steve Loughran added a comment - 17/Jun/14 17:55 HADOOP-9555 updates ZK to 3.4.6

Hide

Permalink

Steve Loughran added a comment - 07/Nov/14 09:13

~~HDFS-7376~~ proposes upgrading jsch to avoid java7 problems

Show

Steve Loughran added a comment - 07/Nov/14 09:13 HDFS-7376 proposes upgrading jsch to avoid java7 problems

Hide

Permalink

Prasanth Jayachandran added a comment - 29/Jul/15 01:43

Steve Loughran With ~~HIVE-11304~~ I am trying to migrate to log4j2.x from log4j1.x. Any plans for the same in hadoop?

Show

Prasanth Jayachandran added a comment - 29/Jul/15 01:43 Steve Loughran With HIVE-11304 I am trying to migrate to log4j2.x from log4j1.x. Any plans for the same in hadoop?

Hide

Permalink

Steve Loughran added a comment - 29/Jul/15 04:31

Hadoop only explicitly gets at log4j in some tests, so code-wise, excluding those, switching to log4j v 2 is technically a matter of changing the mvn version.

Except

we don't know what happens downstream
logging is essential to all services
rolling logs of YARN services matters too.

For those reasons, it's hard to see enthusiasm for a change. Especially as async logging has a price: when things fail the log may not be complete.

What is possible is to move the hadoop logging to slf4j off commons logging; that's something that we'd like to see in new code, even though migrating old code is TBD. Do that and switching to log4j or even logback can be done by changing the slf4j dependency

Show

Steve Loughran added a comment - 29/Jul/15 04:31 Hadoop only explicitly gets at log4j in some tests, so code-wise, excluding those, switching to log4j v 2 is technically a matter of changing the mvn version. Except we don't know what happens downstream logging is essential to all services rolling logs of YARN services matters too. For those reasons, it's hard to see enthusiasm for a change. Especially as async logging has a price: when things fail the log may not be complete. What is possible is to move the hadoop logging to slf4j off commons logging; that's something that we'd like to see in new code, even though migrating old code is TBD. Do that and switching to log4j or even logback can be done by changing the slf4j dependency

Hide

Permalink

ASF GitHub Bot added a comment - 16/Mar/16 11:28

Github user steveloughran commented on the pull request:

https://github.com/apache/hadoop/pull/84#issuecomment-197274477

Please can you file a separate JIRA for these two changes, link to HADOOP-9991 and include justification. Version updates are a traumatic issue in Hadoop and done fairly reluctantly.

FWIW, updating netty-all from beta to final makes sense just from a release perspective; updating the other jetty less so

Show

ASF GitHub Bot added a comment - 16/Mar/16 11:28 Github user steveloughran commented on the pull request: https://github.com/apache/hadoop/pull/84#issuecomment-197274477 Please can you file a separate JIRA for these two changes, link to HADOOP-9991 and include justification. Version updates are a traumatic issue in Hadoop and done fairly reluctantly. FWIW, updating netty-all from beta to final makes sense just from a release perspective; updating the other jetty less so

Hide

Permalink

ASF GitHub Bot added a comment - 16/Mar/16 13:17

Github user ceefour commented on the pull request:

https://github.com/apache/hadoop/pull/84#issuecomment-197323519

@steveloughran This pull request now only updates netty-all (minor version update).

Show

ASF GitHub Bot added a comment - 16/Mar/16 13:17 Github user ceefour commented on the pull request: https://github.com/apache/hadoop/pull/84#issuecomment-197323519 @steveloughran This pull request now only updates netty-all (minor version update).

Hide

Permalink

ASF GitHub Bot added a comment - 16/Mar/16 13:22

GitHub user ceefour opened a pull request:

https://github.com/apache/hadoop/pull/85

HADOOP-9991. Update netty to 3.7.1.Final because hadoop-client 2.7.2

depends on zookeeper 3.4.6 which depends on netty 3.7.x. Related to
https://github.com/apache/hadoop/pull/84

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ceefour/hadoop HADOOP-9991-netty371

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hadoop/pull/85.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #85

commit 2a2c248f8516e69c8f8bfe979765f597cb1c8729
Author: Hendy Irawan <ceefour666@gmail.com>
Date: 2016-03-16T13:18:56Z

HADOOP-9991. Update netty to 3.7.1.Final because hadoop-client 2.7.2
depends on zookeeper 3.4.6 which depends on netty 3.7.x. Related to
https://github.com/apache/hadoop/pull/84

Show

ASF GitHub Bot added a comment - 16/Mar/16 13:22 GitHub user ceefour opened a pull request: https://github.com/apache/hadoop/pull/85 HADOOP-9991 . Update netty to 3.7.1.Final because hadoop-client 2.7.2 depends on zookeeper 3.4.6 which depends on netty 3.7.x. Related to https://github.com/apache/hadoop/pull/84 You can merge this pull request into a Git repository by running: $ git pull https://github.com/ceefour/hadoop HADOOP-9991 -netty371 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/85.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #85 commit 2a2c248f8516e69c8f8bfe979765f597cb1c8729 Author: Hendy Irawan <ceefour666@gmail.com> Date: 2016-03-16T13:18:56Z HADOOP-9991 . Update netty to 3.7.1.Final because hadoop-client 2.7.2 depends on zookeeper 3.4.6 which depends on netty 3.7.x. Related to https://github.com/apache/hadoop/pull/84

Hide

Permalink

ASF GitHub Bot added a comment - 16/Mar/16 13:49

Github user steveloughran commented on the pull request:

https://github.com/apache/hadoop/pull/84#issuecomment-197340633

sorry, I should have been clearer
-create a new Hadoop JIRA for this, mark it as a dependency of HADOOP-9991. thanks

Show

ASF GitHub Bot added a comment - 16/Mar/16 13:49 Github user steveloughran commented on the pull request: https://github.com/apache/hadoop/pull/84#issuecomment-197340633 sorry, I should have been clearer -create a new Hadoop JIRA for this, mark it as a dependency of HADOOP-9991 . thanks

Hide

Permalink

ASF GitHub Bot added a comment - 16/Mar/16 14:39

Github user ceefour commented on the pull request:

https://github.com/apache/hadoop/pull/84#issuecomment-197358937

@steveloughran created ~~HADOOP-12927~~

Show

ASF GitHub Bot added a comment - 16/Mar/16 14:39 Github user ceefour commented on the pull request: https://github.com/apache/hadoop/pull/84#issuecomment-197358937 @steveloughran created HADOOP-12927

Hide

Permalink

ASF GitHub Bot added a comment - 18/Mar/16 19:42

Github user ceefour closed the pull request at:

https://github.com/apache/zookeeper/pull/57

Show

ASF GitHub Bot added a comment - 18/Mar/16 19:42 Github user ceefour closed the pull request at: https://github.com/apache/zookeeper/pull/57

People

Assignee:

Steve Loughran

Reporter:

Steve Loughran

Votes:

5 Vote for this issue

Watchers:

37 Start watching this issue

Dates

Created:

23/Sep/13 08:46

Updated:

Tuesday 18:52

Development

Agile

View on Board