Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance
  3. HADOOP-13692

hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts.

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      If an end user's application has a dependency on hadoop-aws and no other Hadoop artifacts, then it picks up a transitive dependency on Jackson 2.5.3 jars through the AWS SDK. This can cause conflicts at deployment time, because Hadoop has a dependency on version 2.2.3, and the 2 versions are not compatible with one another. We can prevent this problem by changing hadoop-aws to declare explicit dependencies on the Jackson artifacts, at the version Hadoop wants.

        Activity

        Hide
        cnauroth Chris Nauroth added a comment -

        I'm attaching patch 001. We can solve the problem simply by listing the Jackson 2 dependencies in hadoop-aws/pom.xml. See below for a before and after comparison of the dependency tree for an application with a dependency on hadoop-aws. Before, you'll see jackson-databind 2.5.3, coming in transitively through aws-java-sdk-core. After, you'll see jackson-databind 2.2.3, coming in transitively as a direct dependency of hadoop-aws.

        For testing, I completed a full test run against an S3 bucket in US-west-2. I also tested some S3A operations through a throw-away client application with a dependency on hadoop-aws.

        We have a similar dependency management challenge for joda-time, but that one is already listed in hadoop-aws/pom.xml, so we don't need to do anything about it in this patch.

        Before

        [INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ test-hadoop-cloud-storage ---
        [INFO] cnauroth:test-hadoop-cloud-storage:jar:0.0.1-SNAPSHOT
        [INFO] \- org.apache.hadoop:hadoop-aws:jar:2.9.0-SNAPSHOT:compile
        [INFO]    +- org.apache.hadoop:hadoop-common:jar:2.9.0-SNAPSHOT:compile
        [INFO]    |  +- org.apache.hadoop:hadoop-annotations:jar:2.9.0-SNAPSHOT:compile
        [INFO]    |  |  \- jdk.tools:jdk.tools:jar:1.8:system
        [INFO]    |  +- com.google.guava:guava:jar:11.0.2:compile
        [INFO]    |  +- commons-cli:commons-cli:jar:1.2:compile
        [INFO]    |  +- org.apache.commons:commons-math3:jar:3.1.1:compile
        [INFO]    |  +- xmlenc:xmlenc:jar:0.52:compile
        [INFO]    |  +- org.apache.httpcomponents:httpclient:jar:4.5.2:compile
        [INFO]    |  |  \- org.apache.httpcomponents:httpcore:jar:4.4.4:compile
        [INFO]    |  +- commons-codec:commons-codec:jar:1.4:compile
        [INFO]    |  +- commons-io:commons-io:jar:2.4:compile
        [INFO]    |  +- commons-net:commons-net:jar:3.1:compile
        [INFO]    |  +- commons-collections:commons-collections:jar:3.2.2:compile
        [INFO]    |  +- javax.servlet:servlet-api:jar:2.5:compile
        [INFO]    |  +- org.mortbay.jetty:jetty:jar:6.1.26:compile
        [INFO]    |  +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
        [INFO]    |  +- org.mortbay.jetty:jetty-sslengine:jar:6.1.26:compile
        [INFO]    |  +- javax.servlet.jsp:jsp-api:jar:2.1:runtime
        [INFO]    |  +- com.sun.jersey:jersey-core:jar:1.9:compile
        [INFO]    |  +- com.sun.jersey:jersey-json:jar:1.9:compile
        [INFO]    |  |  +- org.codehaus.jettison:jettison:jar:1.1:compile
        [INFO]    |  |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
        [INFO]    |  |  |  \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
        [INFO]    |  |  |     +- javax.xml.stream:stax-api:jar:1.0-2:compile
        [INFO]    |  |  |     \- javax.activation:activation:jar:1.1:compile
        [INFO]    |  |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile
        [INFO]    |  |  \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile
        [INFO]    |  +- com.sun.jersey:jersey-server:jar:1.9:compile
        [INFO]    |  |  \- asm:asm:jar:3.1:compile
        [INFO]    |  +- commons-logging:commons-logging:jar:1.1.3:compile
        [INFO]    |  +- log4j:log4j:jar:1.2.17:compile
        [INFO]    |  +- net.java.dev.jets3t:jets3t:jar:0.9.0:compile
        [INFO]    |  |  \- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:compile
        [INFO]    |  +- commons-lang:commons-lang:jar:2.6:compile
        [INFO]    |  +- commons-configuration:commons-configuration:jar:1.6:compile
        [INFO]    |  |  +- commons-digester:commons-digester:jar:1.8:compile
        [INFO]    |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
        [INFO]    |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
        [INFO]    |  +- org.slf4j:slf4j-api:jar:1.7.10:compile
        [INFO]    |  +- org.slf4j:slf4j-log4j12:jar:1.7.10:compile
        [INFO]    |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
        [INFO]    |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
        [INFO]    |  +- org.apache.avro:avro:jar:1.7.4:compile
        [INFO]    |  |  +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
        [INFO]    |  |  \- org.xerial.snappy:snappy-java:jar:1.0.4.1:compile
        [INFO]    |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
        [INFO]    |  +- com.google.code.gson:gson:jar:2.2.4:compile
        [INFO]    |  +- org.apache.hadoop:hadoop-auth:jar:2.9.0-SNAPSHOT:compile
        [INFO]    |  |  +- com.nimbusds:nimbus-jose-jwt:jar:3.9:compile
        [INFO]    |  |  |  +- net.jcip:jcip-annotations:jar:1.0:compile
        [INFO]    |  |  |  \- net.minidev:json-smart:jar:1.1.1:compile
        [INFO]    |  |  +- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
        [INFO]    |  |  |  +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
        [INFO]    |  |  |  +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
        [INFO]    |  |  |  \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
        [INFO]    |  |  \- org.apache.curator:curator-framework:jar:2.7.1:compile
        [INFO]    |  +- com.jcraft:jsch:jar:0.1.51:compile
        [INFO]    |  +- org.apache.curator:curator-client:jar:2.7.1:compile
        [INFO]    |  +- org.apache.curator:curator-recipes:jar:2.7.1:compile
        [INFO]    |  +- com.google.code.findbugs:jsr305:jar:3.0.0:compile
        [INFO]    |  +- org.apache.htrace:htrace-core4:jar:4.0.1-incubating:compile
        [INFO]    |  +- org.apache.zookeeper:zookeeper:jar:3.4.6:compile
        [INFO]    |  |  \- io.netty:netty:jar:3.7.0.Final:compile
        [INFO]    |  \- org.apache.commons:commons-compress:jar:1.4.1:compile
        [INFO]    |     \- org.tukaani:xz:jar:1.0:compile
        [INFO]    +- com.amazonaws:aws-java-sdk-s3:jar:1.10.6:compile
        [INFO]    |  +- com.amazonaws:aws-java-sdk-kms:jar:1.10.6:compile
        [INFO]    |  \- com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile
        [INFO]    |     \- com.fasterxml.jackson.core:jackson-databind:jar:2.5.3:compile
        [INFO]    |        +- com.fasterxml.jackson.core:jackson-annotations:jar:2.5.0:compile
        [INFO]    |        \- com.fasterxml.jackson.core:jackson-core:jar:2.5.3:compile
        [INFO]    \- joda-time:joda-time:jar:2.9.4:compile
        

        After

        [INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ test-hadoop-cloud-storage ---
        [INFO] cnauroth:test-hadoop-cloud-storage:jar:0.0.1-SNAPSHOT
        [INFO] \- org.apache.hadoop:hadoop-aws:jar:2.9.0-SNAPSHOT:compile
        [INFO]    +- org.apache.hadoop:hadoop-common:jar:2.9.0-SNAPSHOT:compile
        [INFO]    |  +- org.apache.hadoop:hadoop-annotations:jar:2.9.0-SNAPSHOT:compile
        [INFO]    |  |  \- jdk.tools:jdk.tools:jar:1.8:system
        [INFO]    |  +- com.google.guava:guava:jar:11.0.2:compile
        [INFO]    |  +- commons-cli:commons-cli:jar:1.2:compile
        [INFO]    |  +- org.apache.commons:commons-math3:jar:3.1.1:compile
        [INFO]    |  +- xmlenc:xmlenc:jar:0.52:compile
        [INFO]    |  +- org.apache.httpcomponents:httpclient:jar:4.5.2:compile
        [INFO]    |  |  \- org.apache.httpcomponents:httpcore:jar:4.4.4:compile
        [INFO]    |  +- commons-codec:commons-codec:jar:1.4:compile
        [INFO]    |  +- commons-io:commons-io:jar:2.4:compile
        [INFO]    |  +- commons-net:commons-net:jar:3.1:compile
        [INFO]    |  +- commons-collections:commons-collections:jar:3.2.2:compile
        [INFO]    |  +- javax.servlet:servlet-api:jar:2.5:compile
        [INFO]    |  +- org.mortbay.jetty:jetty:jar:6.1.26:compile
        [INFO]    |  +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
        [INFO]    |  +- org.mortbay.jetty:jetty-sslengine:jar:6.1.26:compile
        [INFO]    |  +- javax.servlet.jsp:jsp-api:jar:2.1:runtime
        [INFO]    |  +- com.sun.jersey:jersey-core:jar:1.9:compile
        [INFO]    |  +- com.sun.jersey:jersey-json:jar:1.9:compile
        [INFO]    |  |  +- org.codehaus.jettison:jettison:jar:1.1:compile
        [INFO]    |  |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
        [INFO]    |  |  |  \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
        [INFO]    |  |  |     +- javax.xml.stream:stax-api:jar:1.0-2:compile
        [INFO]    |  |  |     \- javax.activation:activation:jar:1.1:compile
        [INFO]    |  |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile
        [INFO]    |  |  \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile
        [INFO]    |  +- com.sun.jersey:jersey-server:jar:1.9:compile
        [INFO]    |  |  \- asm:asm:jar:3.1:compile
        [INFO]    |  +- commons-logging:commons-logging:jar:1.1.3:compile
        [INFO]    |  +- log4j:log4j:jar:1.2.17:compile
        [INFO]    |  +- net.java.dev.jets3t:jets3t:jar:0.9.0:compile
        [INFO]    |  |  \- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:compile
        [INFO]    |  +- commons-lang:commons-lang:jar:2.6:compile
        [INFO]    |  +- commons-configuration:commons-configuration:jar:1.6:compile
        [INFO]    |  |  +- commons-digester:commons-digester:jar:1.8:compile
        [INFO]    |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
        [INFO]    |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
        [INFO]    |  +- org.slf4j:slf4j-api:jar:1.7.10:compile
        [INFO]    |  +- org.slf4j:slf4j-log4j12:jar:1.7.10:compile
        [INFO]    |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
        [INFO]    |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
        [INFO]    |  +- org.apache.avro:avro:jar:1.7.4:compile
        [INFO]    |  |  +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
        [INFO]    |  |  \- org.xerial.snappy:snappy-java:jar:1.0.4.1:compile
        [INFO]    |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
        [INFO]    |  +- com.google.code.gson:gson:jar:2.2.4:compile
        [INFO]    |  +- org.apache.hadoop:hadoop-auth:jar:2.9.0-SNAPSHOT:compile
        [INFO]    |  |  +- com.nimbusds:nimbus-jose-jwt:jar:3.9:compile
        [INFO]    |  |  |  +- net.jcip:jcip-annotations:jar:1.0:compile
        [INFO]    |  |  |  \- net.minidev:json-smart:jar:1.1.1:compile
        [INFO]    |  |  +- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
        [INFO]    |  |  |  +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
        [INFO]    |  |  |  +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
        [INFO]    |  |  |  \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
        [INFO]    |  |  \- org.apache.curator:curator-framework:jar:2.7.1:compile
        [INFO]    |  +- com.jcraft:jsch:jar:0.1.51:compile
        [INFO]    |  +- org.apache.curator:curator-client:jar:2.7.1:compile
        [INFO]    |  +- org.apache.curator:curator-recipes:jar:2.7.1:compile
        [INFO]    |  +- com.google.code.findbugs:jsr305:jar:3.0.0:compile
        [INFO]    |  +- org.apache.htrace:htrace-core4:jar:4.0.1-incubating:compile
        [INFO]    |  +- org.apache.zookeeper:zookeeper:jar:3.4.6:compile
        [INFO]    |  |  \- io.netty:netty:jar:3.7.0.Final:compile
        [INFO]    |  \- org.apache.commons:commons-compress:jar:1.4.1:compile
        [INFO]    |     \- org.tukaani:xz:jar:1.0:compile
        [INFO]    +- com.amazonaws:aws-java-sdk-s3:jar:1.10.6:compile
        [INFO]    |  +- com.amazonaws:aws-java-sdk-kms:jar:1.10.6:compile
        [INFO]    |  \- com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile
        [INFO]    +- com.fasterxml.jackson.core:jackson-core:jar:2.2.3:compile
        [INFO]    +- com.fasterxml.jackson.core:jackson-databind:jar:2.2.3:compile
        [INFO]    +- com.fasterxml.jackson.core:jackson-annotations:jar:2.2.3:compile
        [INFO]    \- joda-time:joda-time:jar:2.9.4:compile
        
        Show
        cnauroth Chris Nauroth added a comment - I'm attaching patch 001. We can solve the problem simply by listing the Jackson 2 dependencies in hadoop-aws/pom.xml. See below for a before and after comparison of the dependency tree for an application with a dependency on hadoop-aws. Before, you'll see jackson-databind 2.5.3, coming in transitively through aws-java-sdk-core. After, you'll see jackson-databind 2.2.3, coming in transitively as a direct dependency of hadoop-aws. For testing, I completed a full test run against an S3 bucket in US-west-2. I also tested some S3A operations through a throw-away client application with a dependency on hadoop-aws. We have a similar dependency management challenge for joda-time, but that one is already listed in hadoop-aws/pom.xml, so we don't need to do anything about it in this patch. Before [INFO] --- maven-dependency-plugin:2.8:tree ( default -cli) @ test-hadoop-cloud-storage --- [INFO] cnauroth:test-hadoop-cloud-storage:jar:0.0.1-SNAPSHOT [INFO] \- org.apache.hadoop:hadoop-aws:jar:2.9.0-SNAPSHOT:compile [INFO] +- org.apache.hadoop:hadoop-common:jar:2.9.0-SNAPSHOT:compile [INFO] | +- org.apache.hadoop:hadoop-annotations:jar:2.9.0-SNAPSHOT:compile [INFO] | | \- jdk.tools:jdk.tools:jar:1.8:system [INFO] | +- com.google.guava:guava:jar:11.0.2:compile [INFO] | +- commons-cli:commons-cli:jar:1.2:compile [INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:compile [INFO] | +- xmlenc:xmlenc:jar:0.52:compile [INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.2:compile [INFO] | | \- org.apache.httpcomponents:httpcore:jar:4.4.4:compile [INFO] | +- commons-codec:commons-codec:jar:1.4:compile [INFO] | +- commons-io:commons-io:jar:2.4:compile [INFO] | +- commons-net:commons-net:jar:3.1:compile [INFO] | +- commons-collections:commons-collections:jar:3.2.2:compile [INFO] | +- javax.servlet:servlet-api:jar:2.5:compile [INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:compile [INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile [INFO] | +- org.mortbay.jetty:jetty-sslengine:jar:6.1.26:compile [INFO] | +- javax.servlet.jsp:jsp-api:jar:2.1:runtime [INFO] | +- com.sun.jersey:jersey-core:jar:1.9:compile [INFO] | +- com.sun.jersey:jersey-json:jar:1.9:compile [INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:compile [INFO] | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile [INFO] | | | \- javax.xml.bind:jaxb-api:jar:2.2.2:compile [INFO] | | | +- javax.xml.stream:stax-api:jar:1.0-2:compile [INFO] | | | \- javax.activation:activation:jar:1.1:compile [INFO] | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile [INFO] | | \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile [INFO] | +- com.sun.jersey:jersey-server:jar:1.9:compile [INFO] | | \- asm:asm:jar:3.1:compile [INFO] | +- commons-logging:commons-logging:jar:1.1.3:compile [INFO] | +- log4j:log4j:jar:1.2.17:compile [INFO] | +- net.java.dev.jets3t:jets3t:jar:0.9.0:compile [INFO] | | \- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:compile [INFO] | +- commons-lang:commons-lang:jar:2.6:compile [INFO] | +- commons-configuration:commons-configuration:jar:1.6:compile [INFO] | | +- commons-digester:commons-digester:jar:1.8:compile [INFO] | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile [INFO] | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile [INFO] | +- org.slf4j:slf4j-api:jar:1.7.10:compile [INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.10:compile [INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile [INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile [INFO] | +- org.apache.avro:avro:jar:1.7.4:compile [INFO] | | +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile [INFO] | | \- org.xerial.snappy:snappy-java:jar:1.0.4.1:compile [INFO] | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile [INFO] | +- com.google.code.gson:gson:jar:2.2.4:compile [INFO] | +- org.apache.hadoop:hadoop-auth:jar:2.9.0-SNAPSHOT:compile [INFO] | | +- com.nimbusds:nimbus-jose-jwt:jar:3.9:compile [INFO] | | | +- net.jcip:jcip-annotations:jar:1.0:compile [INFO] | | | \- net.minidev:json-smart:jar:1.1.1:compile [INFO] | | +- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile [INFO] | | | +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile [INFO] | | | +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile [INFO] | | | \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile [INFO] | | \- org.apache.curator:curator-framework:jar:2.7.1:compile [INFO] | +- com.jcraft:jsch:jar:0.1.51:compile [INFO] | +- org.apache.curator:curator-client:jar:2.7.1:compile [INFO] | +- org.apache.curator:curator-recipes:jar:2.7.1:compile [INFO] | +- com.google.code.findbugs:jsr305:jar:3.0.0:compile [INFO] | +- org.apache.htrace:htrace-core4:jar:4.0.1-incubating:compile [INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.6:compile [INFO] | | \- io.netty:netty:jar:3.7.0.Final:compile [INFO] | \- org.apache.commons:commons-compress:jar:1.4.1:compile [INFO] | \- org.tukaani:xz:jar:1.0:compile [INFO] +- com.amazonaws:aws-java-sdk-s3:jar:1.10.6:compile [INFO] | +- com.amazonaws:aws-java-sdk-kms:jar:1.10.6:compile [INFO] | \- com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile [INFO] | \- com.fasterxml.jackson.core:jackson-databind:jar:2.5.3:compile [INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.5.0:compile [INFO] | \- com.fasterxml.jackson.core:jackson-core:jar:2.5.3:compile [INFO] \- joda-time:joda-time:jar:2.9.4:compile After [INFO] --- maven-dependency-plugin:2.8:tree ( default -cli) @ test-hadoop-cloud-storage --- [INFO] cnauroth:test-hadoop-cloud-storage:jar:0.0.1-SNAPSHOT [INFO] \- org.apache.hadoop:hadoop-aws:jar:2.9.0-SNAPSHOT:compile [INFO] +- org.apache.hadoop:hadoop-common:jar:2.9.0-SNAPSHOT:compile [INFO] | +- org.apache.hadoop:hadoop-annotations:jar:2.9.0-SNAPSHOT:compile [INFO] | | \- jdk.tools:jdk.tools:jar:1.8:system [INFO] | +- com.google.guava:guava:jar:11.0.2:compile [INFO] | +- commons-cli:commons-cli:jar:1.2:compile [INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:compile [INFO] | +- xmlenc:xmlenc:jar:0.52:compile [INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.2:compile [INFO] | | \- org.apache.httpcomponents:httpcore:jar:4.4.4:compile [INFO] | +- commons-codec:commons-codec:jar:1.4:compile [INFO] | +- commons-io:commons-io:jar:2.4:compile [INFO] | +- commons-net:commons-net:jar:3.1:compile [INFO] | +- commons-collections:commons-collections:jar:3.2.2:compile [INFO] | +- javax.servlet:servlet-api:jar:2.5:compile [INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:compile [INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile [INFO] | +- org.mortbay.jetty:jetty-sslengine:jar:6.1.26:compile [INFO] | +- javax.servlet.jsp:jsp-api:jar:2.1:runtime [INFO] | +- com.sun.jersey:jersey-core:jar:1.9:compile [INFO] | +- com.sun.jersey:jersey-json:jar:1.9:compile [INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:compile [INFO] | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile [INFO] | | | \- javax.xml.bind:jaxb-api:jar:2.2.2:compile [INFO] | | | +- javax.xml.stream:stax-api:jar:1.0-2:compile [INFO] | | | \- javax.activation:activation:jar:1.1:compile [INFO] | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile [INFO] | | \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile [INFO] | +- com.sun.jersey:jersey-server:jar:1.9:compile [INFO] | | \- asm:asm:jar:3.1:compile [INFO] | +- commons-logging:commons-logging:jar:1.1.3:compile [INFO] | +- log4j:log4j:jar:1.2.17:compile [INFO] | +- net.java.dev.jets3t:jets3t:jar:0.9.0:compile [INFO] | | \- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:compile [INFO] | +- commons-lang:commons-lang:jar:2.6:compile [INFO] | +- commons-configuration:commons-configuration:jar:1.6:compile [INFO] | | +- commons-digester:commons-digester:jar:1.8:compile [INFO] | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile [INFO] | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile [INFO] | +- org.slf4j:slf4j-api:jar:1.7.10:compile [INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.10:compile [INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile [INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile [INFO] | +- org.apache.avro:avro:jar:1.7.4:compile [INFO] | | +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile [INFO] | | \- org.xerial.snappy:snappy-java:jar:1.0.4.1:compile [INFO] | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile [INFO] | +- com.google.code.gson:gson:jar:2.2.4:compile [INFO] | +- org.apache.hadoop:hadoop-auth:jar:2.9.0-SNAPSHOT:compile [INFO] | | +- com.nimbusds:nimbus-jose-jwt:jar:3.9:compile [INFO] | | | +- net.jcip:jcip-annotations:jar:1.0:compile [INFO] | | | \- net.minidev:json-smart:jar:1.1.1:compile [INFO] | | +- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile [INFO] | | | +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile [INFO] | | | +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile [INFO] | | | \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile [INFO] | | \- org.apache.curator:curator-framework:jar:2.7.1:compile [INFO] | +- com.jcraft:jsch:jar:0.1.51:compile [INFO] | +- org.apache.curator:curator-client:jar:2.7.1:compile [INFO] | +- org.apache.curator:curator-recipes:jar:2.7.1:compile [INFO] | +- com.google.code.findbugs:jsr305:jar:3.0.0:compile [INFO] | +- org.apache.htrace:htrace-core4:jar:4.0.1-incubating:compile [INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.6:compile [INFO] | | \- io.netty:netty:jar:3.7.0.Final:compile [INFO] | \- org.apache.commons:commons-compress:jar:1.4.1:compile [INFO] | \- org.tukaani:xz:jar:1.0:compile [INFO] +- com.amazonaws:aws-java-sdk-s3:jar:1.10.6:compile [INFO] | +- com.amazonaws:aws-java-sdk-kms:jar:1.10.6:compile [INFO] | \- com.amazonaws:aws-java-sdk-core:jar:1.10.6:compile [INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.2.3:compile [INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.2.3:compile [INFO] +- com.fasterxml.jackson.core:jackson-annotations:jar:2.2.3:compile [INFO] \- joda-time:joda-time:jar:2.9.4:compile
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 0s branch-2 passed
        +1 compile 0m 22s branch-2 passed with JDK v1.8.0_101
        +1 compile 0m 19s branch-2 passed with JDK v1.7.0_111
        +1 mvnsite 0m 25s branch-2 passed
        +1 mvneclipse 0m 21s branch-2 passed
        +1 javadoc 0m 14s branch-2 passed with JDK v1.8.0_101
        +1 javadoc 0m 15s branch-2 passed with JDK v1.7.0_111
        +1 mvninstall 0m 17s the patch passed
        +1 compile 0m 14s the patch passed with JDK v1.8.0_101
        +1 javac 0m 14s the patch passed
        +1 compile 0m 16s the patch passed with JDK v1.7.0_111
        +1 javac 0m 16s the patch passed
        +1 mvnsite 0m 22s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 javadoc 0m 11s the patch passed with JDK v1.8.0_101
        +1 javadoc 0m 13s the patch passed with JDK v1.7.0_111
        +1 unit 0m 19s hadoop-aws in the patch passed with JDK v1.7.0_111.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        12m 52s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:b59b8b7
        JIRA Issue HADOOP-13692
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832060/HADOOP-13692-branch-2.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 866f97fe5d35 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / 9315242
        Default Java 1.7.0_111
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
        JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10697/testReport/
        modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10697/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 0s branch-2 passed +1 compile 0m 22s branch-2 passed with JDK v1.8.0_101 +1 compile 0m 19s branch-2 passed with JDK v1.7.0_111 +1 mvnsite 0m 25s branch-2 passed +1 mvneclipse 0m 21s branch-2 passed +1 javadoc 0m 14s branch-2 passed with JDK v1.8.0_101 +1 javadoc 0m 15s branch-2 passed with JDK v1.7.0_111 +1 mvninstall 0m 17s the patch passed +1 compile 0m 14s the patch passed with JDK v1.8.0_101 +1 javac 0m 14s the patch passed +1 compile 0m 16s the patch passed with JDK v1.7.0_111 +1 javac 0m 16s the patch passed +1 mvnsite 0m 22s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 11s the patch passed with JDK v1.8.0_101 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_111 +1 unit 0m 19s hadoop-aws in the patch passed with JDK v1.7.0_111. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 12m 52s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HADOOP-13692 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832060/HADOOP-13692-branch-2.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 866f97fe5d35 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 9315242 Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10697/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10697/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        cnauroth Chris Nauroth added a comment -

        The pre-commit test4tests warning isn't actionable, because this is a build change only. I tested manually.

        Show
        cnauroth Chris Nauroth added a comment - The pre-commit test4tests warning isn't actionable, because this is a build change only. I tested manually.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        Looking at the spark stuff, I'm explicitly adding in (and blaming on aws SDK 10.6+), another jackson entry

              <dependency>
                <groupId>com.fasterxml.jackson.dataformat</groupId>
                <artifactId>jackson-dataformat-cbor</artifactId>
                <version>${fasterxml.jackson.version}</version>
              </dependency>
        

        Looking into HADOOP-13050; looks like it comes in after AWS SDK 10.6, so not relevant.

        +1

        Show
        stevel@apache.org Steve Loughran added a comment - Looking at the spark stuff, I'm explicitly adding in (and blaming on aws SDK 10.6+), another jackson entry <dependency> <groupId>com.fasterxml.jackson.dataformat</groupId> <artifactId>jackson-dataformat-cbor</artifactId> <version>${fasterxml.jackson.version}</version> </dependency> Looking into HADOOP-13050 ; looks like it comes in after AWS SDK 10.6, so not relevant. +1
        Hide
        cnauroth Chris Nauroth added a comment -

        Thank you, Steve. I committed this to trunk, branch-2 and branch-2.8.

        Show
        cnauroth Chris Nauroth added a comment - Thank you, Steve. I committed this to trunk, branch-2 and branch-2.8.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10567 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10567/)
        HADOOP-13692. hadoop-aws should declare explicit dependency on Jackson 2 (cnauroth: rev 69620f955997250d1b543d86d4907ee50218152a)

        • (edit) hadoop-tools/hadoop-aws/pom.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10567 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10567/ ) HADOOP-13692 . hadoop-aws should declare explicit dependency on Jackson 2 (cnauroth: rev 69620f955997250d1b543d86d4907ee50218152a) (edit) hadoop-tools/hadoop-aws/pom.xml
        Hide
        stevel@apache.org Steve Loughran added a comment -

        just an FYI, this broke my own unit tests on the SPARK-1481 branch

        2016-10-13 18:51:13,888 [ScalaTest-main] INFO  cloud.CloudSuite (Logging.scala:logInfo(54)) - Loading configuration from ../../cloud.xml
        2016-10-13 18:51:14,214 [ScalaTest-main] WARN  util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
        *** RUN ABORTED ***
          java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z
          at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:537)
          at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:448)
          at com.amazonaws.util.json.Jackson.<clinit>(Jackson.java:32)
          at com.amazonaws.internal.config.InternalConfig.loadfrom(InternalConfig.java:230)
          at com.amazonaws.internal.config.InternalConfig.load(InternalConfig.java:247)
          at com.amazonaws.internal.config.InternalConfig$Factory.<clinit>(InternalConfig.java:282)
          at com.amazonaws.util.VersionInfoUtils.userAgent(VersionInfoUtils.java:139)
          at com.amazonaws.util.VersionInfoUtils.initializeUserAgent(VersionInfoUtils.java:134)
          at com.amazonaws.util.VersionInfoUtils.getUserAgent(VersionInfoUtils.java:95)
          at com.amazonaws.ClientConfiguration.<clinit>(ClientConfiguration.java:42)
          ...
        [INFO] ------------------------------------------------------------------------
        [INFO] BUILD FAILURE
        

        But this problem goes away in spark-assembly, the release of spark, etc. Purely this module. Which is why I didn't catch this earlier as the system integration tests were all happy.

        cause:

        1. there's a newer version of jackson in use in spark (2.6.5)
        2. which overrides the declarations of jackson-annotations and jackson-databind under hadoop-aws
        3. and which have transitive dependencies on jackson-common.
        4. the explicitdeclaration of jackson-common has pulled that reference one step up the dependency graph (i.e. from under spark-cloud/hadoop-aws/amazon-aws/jackson-common.jar) to spark-cloud/hadoop-aws/jackson-common.jar.
        5. which gives the hadoop-aws version precedence over the one transitively referenced by the (overridde) jackson-annotations, pulled in directly from spark-core JAR.
        6. so creating a version inconsistency which surfaces during test runs.

        The problem isn't in spark-assembly.jar as it refers to spark-core jar directly, plcking that version up instead.

        Essentially: the fact that maven uses closest-version first in its version resolution policy means that the depth of transitive dependencies controls whether things run or not; the explicit declaration of the dependency was enough to cause this to surface.

        Fix: explicitly exclude the hadoop-aws jackson dependencies, as was already done for hadoop-azure.

        This is not me faulting my own work (how would I!), only showing that you do need to be careful across projects as to what transitive stuff you pull in, as it turns out to be incredibly brittle. We didn't change the jackson version here, only made that choice explicit, and a downstream test suite fails.

        Show
        stevel@apache.org Steve Loughran added a comment - just an FYI, this broke my own unit tests on the SPARK-1481 branch 2016-10-13 18:51:13,888 [ScalaTest-main] INFO cloud.CloudSuite (Logging.scala:logInfo(54)) - Loading configuration from ../../cloud.xml 2016-10-13 18:51:14,214 [ScalaTest-main] WARN util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native -hadoop library for your platform... using builtin-java classes where applicable *** RUN ABORTED *** java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:537) at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:448) at com.amazonaws.util.json.Jackson.<clinit>(Jackson.java:32) at com.amazonaws.internal.config.InternalConfig.loadfrom(InternalConfig.java:230) at com.amazonaws.internal.config.InternalConfig.load(InternalConfig.java:247) at com.amazonaws.internal.config.InternalConfig$Factory.<clinit>(InternalConfig.java:282) at com.amazonaws.util.VersionInfoUtils.userAgent(VersionInfoUtils.java:139) at com.amazonaws.util.VersionInfoUtils.initializeUserAgent(VersionInfoUtils.java:134) at com.amazonaws.util.VersionInfoUtils.getUserAgent(VersionInfoUtils.java:95) at com.amazonaws.ClientConfiguration.<clinit>(ClientConfiguration.java:42) ... [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE But this problem goes away in spark-assembly, the release of spark, etc. Purely this module. Which is why I didn't catch this earlier as the system integration tests were all happy. cause: there's a newer version of jackson in use in spark (2.6.5) which overrides the declarations of jackson-annotations and jackson-databind under hadoop-aws and which have transitive dependencies on jackson-common. the explicitdeclaration of jackson-common has pulled that reference one step up the dependency graph (i.e. from under spark-cloud/hadoop-aws/amazon-aws/jackson-common.jar) to spark-cloud/hadoop-aws/jackson-common.jar. which gives the hadoop-aws version precedence over the one transitively referenced by the (overridde) jackson-annotations, pulled in directly from spark-core JAR. so creating a version inconsistency which surfaces during test runs. The problem isn't in spark-assembly.jar as it refers to spark-core jar directly, plcking that version up instead. Essentially: the fact that maven uses closest-version first in its version resolution policy means that the depth of transitive dependencies controls whether things run or not; the explicit declaration of the dependency was enough to cause this to surface. Fix: explicitly exclude the hadoop-aws jackson dependencies, as was already done for hadoop-azure. This is not me faulting my own work (how would I!), only showing that you do need to be careful across projects as to what transitive stuff you pull in, as it turns out to be incredibly brittle. We didn't change the jackson version here, only made that choice explicit, and a downstream test suite fails.

          People

          • Assignee:
            cnauroth Chris Nauroth
            Reporter:
            cnauroth Chris Nauroth
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development