works-at-all. doesn't filter by IA.Public annotation.

-01

• create shaded artifacts for downstream compile, runtime, and test scopes
• hadoop-client-api is compile scope, currently all org.apache.hadoop from hadoop-client
• hadoop-client-runtime is runtime scope, contains relocated third party deps from hadoop-client
• hadoop-client-minicluster is test scope, has the hadoop-minicluster and deps shaded (w/third party relocated) barring those things found in api and runtime.
• has a local implementation of MSHADE-182
• requires a maven version that can use the dependency-plugin feature of banning all transitive dependencies.

-1 overall

This message was automatically generated.

excellent catch by the QA bot. I had only been doing package / install builds and so didn't notice that enforcer was firing at validate instead of verify.

-02

• bind the enforcer rules to the verify phase

downstream is failing now. need to figure out if I messed something up in my moving commits around branches.

-1 overall

This message was automatically generated.

Updated patch, now that HBase master works with Hadoop branch-2 again.

-03

• includes a local fork of MSHADE-182 (to handle java services reloction)
• properly relocate third party references in the client-api bytecode
• avoid relocating references to JDK packages that are in com.sun
• avoid shaded jersey binding to javax services

Known issues right now

• logging implementations are still relocated
• HTrace is relocated
  • because of how htrace is implemented, this prevents starting a span in a client app and having the client continue with it as a parent span.
• some other HBase second-pass tests fail post-rebase that I am still tracking down
  • Notably, all the HBase MR tests fail because the shaded minicluster YARN has some startup problem related to jersey

submitting patch just to get a QA look at checkstyle / whitespace. the YARN thing needs to be fixed before this should be pushed.

Known issue

• some of the HBase test failures are because jaxb stuff isn't properly hidden in the shaded minicluster

-1 overall

This message was automatically generated.

This is great work! Looking forward to the final patch. I'm testing it on 2.6 and it mostly works fine. A few issues:

1. How can an application specify log4j.properties? I tried using both shaded and unshaded class names as appenders and couldn't get it to work.
2. The Jersey shading is causing problems to WebHDFS. In unit testing my mini DFS cluster cannot start web server. If I use the hadoop-client-runtime jar to talk to a WebHDFS server, the response cannot be correctly parsed. Seems related to ServiceLoader

-04

• update for current Hadoop 3 branch
• clean up for dependencies that are already in Java 7+ SE

limitations:

• logging libraries still relocated
• htrace libraries still relocated
• timeline server excluded from shaded minicluster and marked as optional

I'm vetting this against HBase now, but figured I'd post an update for some initial review.

I think the answer for the logging libraries and htrace is to leave them unshaded, since it's common to want to modify logging settings and to want to trace through e.g. the hdfs client. Would like some feedback here.

The Jersey shading is causing problems to WebHDFS. In unit testing my mini DFS cluster cannot start web server. If I use the hadoop-client-runtime jar to talk to a WebHDFS server, the response cannot be correctly parsed. Seems related to ServiceLoader

I have to chase this down still. Zhe Zhang can you help me out with a step-by-step for reproducing?

-1 overall

This message was automatically generated.

-1 overall

This message was automatically generated.

Thanks for working on this Sean!

Not sure why the patch didn't apply for precommit, since it applied for me locally. Since it modifies hadoop-maven-plugins, I "mvn install"'d it first per the normal build instructions.

Some review comments:

• Typos: "hte" "htey" "dependnecies" "itis"
• I see this comment: skip org.apache.avro:avro-ipc because it doesn't look like hadoop-common actually uses it. If there are other issues like this, it'd be nice to surface them in a JIRA comment or JIRA so we can think about fixing them properly.
• The build failed for me with this error:

[WARNING] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed with message:
Duplicate classes found:

  Found in:
    org.apache.hadoop:hadoop-client-api:jar:3.0.0-alpha2-SNAPSHOT:compile
    org.apache.hadoop:hadoop-client-minicluster:jar:3.0.0-alpha2-SNAPSHOT:compile
  Duplicate classes:
    org/apache/hadoop/ipc/protobuf/TestRpcServiceProtos$TestProtobufRpc2Proto.class
    org/apache/hadoop/ipc/protobuf/TestProtos$EmptyResponseProto$Builder.class
    org/apache/hadoop/ipc/protobuf/TestProtos$1.class
    org/apache/hadoop/ipc/protobuf/TestProtos$SleepResponseProto$1.class
    org/apache/hadoop/ipc/protobuf/TestRpcServiceProtos$OldProtobufRpcProto$Interface.class
    org/apache/hadoop/ipc/protobuf/TestRpcServiceProtos$NewProtobufRpcProto$BlockingInterface.class
    org/apache/hadoop/ipc/protobuf/TestProtos$AuthMethodResponseProto$1.class
...<more proto classes>

trunk is a fast moving target, so if you give me your git hash, I can review against that.

Thanks for the review! I'll update to latest trunk as a part of getting the rest of the feedback addressed.

I'm a bit concerned about the build failure, since even if you review against my current commit (dbd2057), the hygiene check there (that the same class doesn't end up in more than one of the shaded jars) would still have failed later. I'm confused about test protobuf classes showing up in the client-api jar, but hopefully the cause will be obvious once I rebase.

I'm planning to un-shade logging and htrace today, so if you'd prefer I not do that please let me know. A related question, do we have a project-wide logging framework?

Busbey-MBA:hadoop busbey$ git grep -l  "org.slf4j" | grep "\\.java" | wc -l
     320
Busbey-MBA:hadoop busbey$ git grep -l  "org.apache.commons.logging" | grep "\\.java" | wc -l
    1410
Busbey-MBA:hadoop busbey$ git grep -l  "java.util.logging" | grep "\\.java" | wc -l
       2
Busbey-MBA:hadoop busbey$ git grep -l  "org.apache.log4j" | grep "\\.java" | wc -l
     217
Busbey-MBA:hadoop busbey$ git grep -l  "org.apache.logging.log4j" | grep "\\.java" | wc -l
       0

Should I just un-shade all the logging libraries?

Thanks for the quick response Sean,

I'm confused about test protobuf classes showing up in the client-api jar...

Yea, this was a mystery to me too. Could be PEBKAC, in which case I'd appreciate fuller build instructions.

I think the answer for the logging libraries and htrace is to leave them unshaded, since it's common to want to modify logging settings and to want to trace through e.g. the hdfs client. Would like some feedback here.

The logging rationale here sounds good to me. We've been trying to migrate things to slf4j, but evidently we haven't made much progress.

Unshading HTrace also sounds good, since I'm guessing that shading will mess up tracing from an app into the Hadoop client. Colin P. McCabe care to comment more authoritatively?

Thanks for the work Sean Busbey! I just did a quick test with the latest patch.

One high level concern is in terms of maintaining dependencies in the pom's. If a developer adds a new dependency to a module, how would that propagate to these client pom's? Would he/she need to add it to these client pom's for the most part? It wasn't entirely clear to me what that cost of maintenance is. If that is the only way to keep it clean, that's OK. But it would be great if that cost is kept to a minimum.

1.
The patch indeed does not apply for me via plain git apply: it breaks with hadoop-client/pom.xml and hadoop-maven-plugins/pom.xml. I did git apply --reject HADOOP-11804.1.patch.

2.
Once I fixed the git apply issues, I did mvn clean install package -Pdist -DskipTests -Dmaven.javadoc.skip and it fails right away:

[ERROR]   The project org.apache.hadoop:hadoop-client-minicluster:3.0.0-alpha2-SNAPSHOT (/Users/sjlee/git/hadoop-trunk/hadoop-client-modules/hadoop-client-minicluster/pom.xml) has 1 error
[ERROR]     'dependencies.dependency.version' for org.mortbay.jetty:jetty:jar is missing. @ line 266, column 17

I got past it by providing a version for this (chose 6.1.26).

3.
The build still fails with a couple of duplicate classes issues. One is what Andrew reported above. Another is duplicate jetty classes.

[WARNING] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed with message:
Duplicate classes found:

  Found in:
    org.apache.hadoop:hadoop-client-runtime:jar:3.0.0-alpha2-SNAPSHOT:compile
    org.apache.hadoop:hadoop-client-minicluster:jar:3.0.0-alpha2-SNAPSHOT:compile
  Duplicate classes:
    org/apache/hadoop/shaded/org/eclipse/jetty/io/ssl/SslConnection$2.class
    org/apache/hadoop/shaded/org/eclipse/jetty/server/RequestLog.class
    org/apache/hadoop/shaded/org/eclipse/jetty/server/ResourceCache$1.class
    org/apache/hadoop/shaded/org/eclipse/jetty/util/log/AbstractLogger.class
    org/apache/hadoop/shaded/org/eclipse/jetty/util/annotation/Name.class
    org/apache/hadoop/shaded/org/eclipse/jetty/util/component/LifeCycle.class
    org/apache/hadoop/shaded/org/eclipse/jetty/server/HttpChannel$Commit100Callback.class
    org/apache/hadoop/shaded/org/eclipse/jetty/util/ssl/SslContextFactory$1.class
  ...

4.
Was there a significant difficulty in handing the timeline service v.2? Is it just the number of new dependencies we’re pulling in or the fact that there is a HBase dependency?

5.
Regarding the logging libraries, I agree we probably want to exclude them. Things like log4j properties and the way slf4j works can cause issues down the road if shaded.

-05

• update to current master (0e75496)
• includes patch for HADOOP-13789 so precommit can run
• incorporate review feedback
• exclude slf4j backend from hadoop-client pom
• leave runtime dependency on htrace, slf4j-api, commons-logging, and log4j

I made the log4j dependency optional, since there are plenty of use cases where a downstream user wouldn't need it (it's needed for MapReduce and to use some custom Log4j appenders we make).

Yea, this was a mystery to me too. Could be PEBKAC, in which case I'd appreciate fuller build instructions.

I don't know if something changed or if I was just building incorrectly before. When I use the build given by Sangjin Lee I easily reproduced this issue. Since it's a general problem (including test classes in two places) I filed HADOOP-13789 to fix it.

One high level concern is in terms of maintaining dependencies in the pom's. If a developer adds a new dependency to a module, how would that propagate to these client pom's? Would he/she need to add it to these client pom's for the most part? It wasn't entirely clear to me what that cost of maintenance is. If that is the only way to keep it clean, that's OK. But it would be great if that cost is kept to a minimum.

If a developer adds a new dependency that impacts clients, then the only place they should have to update is the shaded minicluster to make sure the dependency is only included in one of the three shaded artifacts. At the moment, this is the only way to make sure a given class only ends up in one place. Ideally we'd reduce the long term cost by adding a custom shader that can exclude classes that appear in a given dependency. I'm not sure how long creating that shader will take, so I didn't do it yet.

1. patch doesn't apply
2. no mortbay.jetty version
3. duplicate jetty classes

These were all fallout from the update to jetty 9. They should all be cleaned up in this rebase.

4. Was there a significant difficulty in handing the timeline service v.2? Is it just the number of new dependencies we’re pulling in or the fact that there is a HBase dependency?

the volume of dependencies caused me some concern, but HBase getting pulled in by default was the blocker for me. Since I'm trying to test HBase as a downstream application having a shaded hbase dependency included (for a feature that is off by default) seems ill advised.

Adding it back in for downstream folks that need it should be straight-forward, since it is included as an optional dependency. they just need to add the artifact as a test dependency.

-1 overall

This message was automatically generated.

updating for precommit feedback now

• It looks like we generate dummy JARs for some of the new modules, since we inherit the jar-plugin executions from the parent pom in hadoop-project-dist. This isn't a big deal, but I have an example for how to disable them at HADOOP-13784.
• We need to update dev-support/bin/dist-layout-stitching so the new artifacts are included in the release layout. You can run maven with "-Pdist" to test this without running the full create-release script.

I'm kicking the tires by trying to move Avro to use the shaded api/runtime artifacts. Compile worked, but the MR tests die with this when trying to make a new JobConf:

testSort(org.apache.avro.mapred.TestWeather)  Time elapsed: 0.011 sec  <<< ERROR!
java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/org/apache/commons/logging/LogFactory
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:197)
        at org.apache.avro.mapred.TestWeather.testSort(TestWeather.java:144)

I looked inside hadoop-client-runtime and don't see commons-logging shaded in there. dependency-tree shows commons-logging being pulled in as a normal dep, which I think is expected.

I can provide my WIP avro patch if you'd like to play with it yourself, but it was pretty trivial.

Andrew Wang: yes, please do not shade HTrace. We have carefully designed HTrace to not have compatibility issues in 4.x (or with old 3.x code), and shading HTrace will prevent spans from being propagated correctly.

-06

• rebased to trunk (7154a20)
• consolidated maven-shade-plugin version
• updated MSHADE-182 implementation for Hadoop checkstyle rules
• updated MSHADE-182 implementation for Hadoop's findbugs configs
• fixed whitespace complaints
• fixed license header on dependency reduced poms (and cleaned up handling of them generally)

The failure in TestQueuingContainerManager was consistent, but unrelated to the patch and rebasing has fixed it.

sorry for the noise, v6 is from yesterday afternoon, apparently I forgot to hit submit. I almost have v7 done now with your current feedback Andrew Wang. the class not found exception is a :facepalm: thing I forgot to update in shading.

-07

• rebased to trunk (dbb133c)
• update to v4 of HADOOP-13789
• prevent spurious jars
• make sure dependencies excluded from shading aren't relocated
• add client jars to dist tarball

Thanks for the rev Sean. I tried it with Avro and got NoClassDefFound for Log4J:

testSort(org.apache.avro.mapred.TestAvroTextSort)  Time elapsed: 0.051 sec  <<< ERROR!
java.lang.NoClassDefFoundError: org/apache/log4j/Level
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:356)
	at org.apache.avro.mapred.TestAvroTextSort.testSort(TestAvroTextSort.java:37)

I think this is expected based on the contents of the hadoop-client-runtime pom.xml, which marks log4j as optional. I manually added this dependency, and then hit this:

testReadAvro(org.apache.avro.hadoop.io.TestAvroSequenceFile)  Time elapsed: 0.016 sec  <<< ERROR!
java.lang.NullPointerException: null
	at org.apache.hadoop.io.serializer.SerializationFactory.<init>(SerializationFactory.java:58)
	at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1248)
	at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1207)
	at org.apache.avro.hadoop.io.AvroSequenceFile$Writer.<init>(AvroSequenceFile.java:532)
	at org.apache.avro.hadoop.io.TestAvroSequenceFile.writeSequenceFile(TestAvroSequenceFile.java:200)
	at org.apache.avro.hadoop.io.TestAvroSequenceFile.testReadAvro(TestAvroSequenceFile.java:53)

I decompiled the SerializationFactory class, and noticed that it messed with the config key. I think we need to add some kind of exclusion for CommonConfigurationKeysPublic.

    // before
    if (conf.get(CommonConfigurationKeys.IO_SERIALIZATIONS_KEY).equals("")) {
    // decompiled
    if (conf.get("org.apache.hadoop.shaded.io.serializations").equals("")) {

Here's my Avro diff for master (without the log4j addition) if you want to try this yourself:

https://gist.github.com/anonymous/c064c283348a2d1bbec00845678339f9

I tried it with Avro and got NoClassDefFound for Log4J

I think this is expected based on the contents of the hadoop-client-runtime pom.xml, which marks log4j as optional.

Yes, that's correct. Since Avro uses JobConf it will have to add a dependency on log4j. I'll make sure to call that out in docs that provide examples of use. If I wanted to try to refactor out the use of log4j in JobConf, do you think that'd be feasible in time for 3.0?

I decompiled the SerializationFactory class, and noticed that it messed with the config key. I think we need to add some kind of exclusion for CommonConfigurationKeysPublic.

    // before
    if (conf.get(CommonConfigurationKeys.IO_SERIALIZATIONS_KEY).equals("")) {
    // decompiled
    if (conf.get("org.apache.hadoop.shaded.io.serializations").equals("")) {

Interesting. Let me figure out a test for this and come up with a fix.

The javac compiler inlines string or integer primitive constants.

right, the problem isn't the inlining, it's that the key has been rewritten to use org.apache.hadoop.shaded as a prefix, which presumably none of our actual configuration files use.

Oh got you. I read it too quickly.

-1 overall

This message was automatically generated.

If I wanted to try to refactor out the use of log4j in JobConf, do you think that'd be feasible in time for 3.0?

That'd be awesome! If this is just a mechanical replacement of log4j API usage with SLF4J, someone might pick it up if you file the JIRA.

odd. the precommit build is marked as aborted and a bunch of the referenced artifacts didn't get archived.

lemme re-run things.

If I wanted to try to refactor out the use of log4j in JobConf, do you think that'd be feasible in time for 3.0?

That'd be awesome! If this is just a mechanical replacement of log4j API usage with SLF4J, someone might pick it up if you file the JIRA.

I didn't look too closely, but I think it's changing how we internally represent default log levels for jobs. I'll see if I can succinctly phrase the request as a JIRA tomorrow.

-1 overall

This message was automatically generated.

-8

• rebased to trunk (e1c6ef2)
• remove patch for HADOOP-13879 now that it's been committed

Still working on the mis-shading mentioned earlier.

-1 overall

This message was automatically generated.

11035 is from another aborted build before the timeout change.

-08

• rebase to trunk (aab9737)
• don't relocate the Hadoop config keys that start with "io." or "net."

I'm going to work through the Avro example tomorrow morning, just want to get this put up for precommit and visibility in the mean time.

I couldn't get an automated test easily. We essentially need an enforcer rule that can get the contents of String constants in a set of class files. I included a comment in the enforce-invariants pom of how I checked this manually. could use some better formatting. Not sure if it's worth pushing this check into dev-support and then invoking ant-run in the verify phase to run it or something like that.

-1 overall

This message was automatically generated.

Lines that start with ????? in the ASF License  report indicate files that do not have an Apache license header:
 !????? /testptch/hadoop/hadoop-build-tools/maven-eclipse.xml
 !????? /testptch/hadoop/hadoop-build-tools/.externalToolBuilders/Maven_Ant_Builder.launch
 !????? hadoop-client/.classpath
 !????? hadoop-client/.project
 !????? hadoop-client/.settings/org.eclipse.jdt.core.prefs

this looks like a gap between our building and yetus. Anyone have an idea of how I get those temporary files to show up? I don't see any of them locally.

In any case, none of these are files from this patch (and the hadoop-client directory shouldn't exist after the patch).

okay, mvn eclipse:eclipse makes them show up. I'm pretty sure we don't run that as a part of the release processs, so these files shouldn't end up published. we should add them to the ignore list for the rat-plugin, but that's not related to this ticket specifically.

-9

• rebased to trunk (b2d4b7b)
• fixed tabs

now back to working on avro.

Okay, I've updated Andrew Wang's patch for avro master: https://gist.github.com/busbey/caccd2fedda36409df4a319deb3e69c2

And given v9 locally installed, I get a clean run of ./build.sh test in lang/java.

the avro-mapred module ends up with this dep tree:

[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ avro-mapred ---
[INFO] org.apache.avro:avro-mapred:bundle:1.9.0-SNAPSHOT
[INFO] +- org.apache.avro:avro-ipc:jar:1.9.0-SNAPSHOT:compile
[INFO] |  +- org.apache.avro:avro:jar:1.9.0-SNAPSHOT:compile
[INFO] |  +- org.mortbay.jetty:jetty:jar:6.1.26:compile
[INFO] |  +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] |  +- io.netty:netty:jar:3.5.13.Final:compile
[INFO] |  +- org.apache.velocity:velocity:jar:1.7:compile
[INFO] |  |  +- commons-collections:commons-collections:jar:3.2.1:compile
[INFO] |  |  \- commons-lang:commons-lang:jar:2.4:compile
[INFO] |  \- org.mortbay.jetty:servlet-api:jar:2.5-20081211:compile
[INFO] +- org.apache.avro:avro-ipc:jar:tests:1.9.0-SNAPSHOT:test
[INFO] +- org.apache.avro:avro:test-jar:tests:1.9.0-SNAPSHOT:test
[INFO] |  +- org.apache.avro:avro-guava-dependencies:jar:1.9.0-SNAPSHOT:compile
[INFO] |  |  +- com.google.guava:guava:jar:11.0.2:compile
[INFO] |  |  \- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
[INFO] |  +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.1.3:compile
[INFO] |  +- org.apache.commons:commons-compress:jar:1.8.1:compile
[INFO] |  \- org.tukaani:xz:jar:1.5:compile
[INFO] +- org.easymock:easymock:jar:3.2:test
[INFO] |  +- cglib:cglib-nodep:jar:2.2.2:test
[INFO] |  \- org.objenesis:objenesis:jar:1.3:test
[INFO] +- org.hamcrest:hamcrest-library:jar:1.3:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] +- commons-codec:commons-codec:jar:1.9:compile
[INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.0.0-alpha2-SNAPSHOT:compile
[INFO] +- org.apache.hadoop:hadoop-client-runtime:jar:3.0.0-alpha2-SNAPSHOT:provided
[INFO] |  +- org.apache.htrace:htrace-core4:jar:4.1.0-incubating:provided
[INFO] |  \- commons-logging:commons-logging:jar:1.1.3:provided
[INFO] +- log4j:log4j:jar:1.2.17:provided
[INFO] +- junit:junit:jar:4.11:test
[INFO] +- org.slf4j:slf4j-api:jar:1.7.7:compile
[INFO] \- org.slf4j:slf4j-simple:jar:1.7.7:runtime

Which looks about as I'd expect (maybe hadoop-client-api should be provided as well, given the notes in the avro pom, but that's probably immaterial).

and avro-tools looks reasonable:

[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ avro-tools ---
[INFO] org.apache.avro:avro-tools:jar:1.9.0-SNAPSHOT
[INFO] +- org.apache.avro:avro:jar:1.9.0-SNAPSHOT:compile
[INFO] |  +- org.apache.avro:avro-guava-dependencies:jar:1.9.0-SNAPSHOT:compile
[INFO] |  |  \- com.google.guava:guava:jar:11.0.2:compile
[INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.1.3:compile
[INFO] |  +- org.apache.commons:commons-compress:jar:1.8.1:compile
[INFO] |  \- org.tukaani:xz:jar:1.5:compile
[INFO] +- org.apache.avro:avro:jar:tests:1.9.0-SNAPSHOT:test
[INFO] +- org.apache.avro:avro-compiler:jar:1.9.0-SNAPSHOT:compile
[INFO] |  +- commons-lang:commons-lang:jar:2.6:compile
[INFO] |  +- org.apache.velocity:velocity:jar:1.7:compile
[INFO] |  |  \- commons-collections:commons-collections:jar:3.2.1:compile
[INFO] |  \- joda-time:joda-time:jar:2.7:compile
[INFO] +- org.apache.avro:avro-ipc:jar:1.9.0-SNAPSHOT:compile
[INFO] |  +- org.mortbay.jetty:jetty:jar:6.1.26:compile
[INFO] |  +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] |  +- io.netty:netty:jar:3.5.13.Final:compile
[INFO] |  \- org.mortbay.jetty:servlet-api:jar:2.5-20081211:compile
[INFO] +- org.apache.avro:avro-mapred:jar:1.9.0-SNAPSHOT:compile
[INFO] |  \- commons-codec:commons-codec:jar:1.9:compile
[INFO] +- org.apache.avro:avro-mapred:test-jar:tests:1.9.0-SNAPSHOT:test
[INFO] +- commons-cli:commons-cli:jar:1.2:compile
[INFO] +- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] +- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] +- org.apache.avro:trevni-core:jar:1.9.0-SNAPSHOT:compile
[INFO] +- org.apache.avro:trevni-avro:jar:1.9.0-SNAPSHOT:compile
[INFO] |  \- org.apache.avro:avro-mapred:jar:hadoop2:1.9.0-SNAPSHOT:compile
[INFO] +- org.apache.avro:trevni-core:jar:tests:1.9.0-SNAPSHOT:compile
[INFO] +- org.apache.avro:trevni-avro:jar:tests:1.9.0-SNAPSHOT:compile
[INFO] +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] +- org.apache.hadoop:hadoop-client-runtime:jar:3.0.0-alpha2-SNAPSHOT:runtime
[INFO] |  \- org.apache.htrace:htrace-core4:jar:4.1.0-incubating:runtime
[INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.0.0-alpha2-SNAPSHOT:compile
[INFO] +- log4j:log4j:jar:1.2.17:compile
[INFO] +- net.sf.jopt-simple:jopt-simple:jar:4.7:compile
[INFO] +- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
[INFO] +- junit:junit:jar:4.11:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- org.slf4j:slf4j-api:jar:1.7.7:compile
[INFO] \- org.slf4j:slf4j-simple:jar:1.7.7:runtime

Do we know yet what the impact is going to be on the hadoop-tools components? Especially with the "on demand" jar detection at build time?

This shouldn't impact the hadoop-tools components AFAICT, since the shaded client jars are opt-in. Only one of the hadoop-tools components uses the hadoop-client dependency (which still works), the others directly reference more specific internal-facing dependencies.

Do you have an example of the kind of problem you're expecting?

-1 overall

This message was automatically generated.

Any feedback Sangjin Lee?

Zhe Zhang any more details here:

The Jersey shading is causing problems to WebHDFS. In unit testing my mini DFS cluster cannot start web server. If I use the hadoop-client-runtime jar to talk to a WebHDFS server, the response cannot be correctly parsed. Seems related to ServiceLoader

I have to chase this down still. Zhe Zhang can you help me out with a step-by-step for reproducing?

I'll take a look at it soon. It would be great if there is a reproducible case that we can easily replicate the problem.

If it is related with the ServiceLoader, you want to look at the properties files from which the ServiceLoader loads classes via reflection (e.g. META-INF/services/org.apache.hadoop.fs.FileSystem). The class names in these files need to match the actual class names in the jar. If the actual class was shaded, the content of these properties files need to be shaded too.

• hadoop-11804-client-test.tar.gz

Including an example program and the steps I used to verify that it can use shaded client/runtime to talk to a WebHDFS instance.

-10

• rebased for trunk (0cfd7ad)
• cleaned up new pom uses of duplicate group ID

ping Sangjin Lee, Zhe Zhang, Andrew Wang.

I think the attached example shows v10 working against a webHDFS instance. Please let me know if there's a different test folks would prefer.

-1 overall

This message was automatically generated.

this weekend I found an issue while setting up some integration tests. should have an update later today.

-11

• adds an integration test for using the shaded minicluster
• fixes some found gaps in dependencies for runtime v minicluster
• fixes a typo in the relocation of javax.servlet for minicluster

known limitation: the integration test I added for webhdfs access currently fails. I believe this is the failure Zhe Zhang was originally referring to. I'm still trying to figure out a workable solution.

-12

• rebase to trunk (c265515)
• updates shaded minicluster to add in the java services declarations we excluded from shaded runtime
• integration test for webhdfs now passes

A limitation of the Java Services API (and that some of the services we have implementations for are part of Java servlets) is that folks using the shaded minicluster for testing will see our shaded implementations for a few APIs.

I believe as of this patch Zhe Zhang's original problem should be solved. The previously attached example should also provide a manual way of verifying running against a live cluster.

-1 overall

This message was automatically generated.

findbugs is a false negative caused by a test-file-only module. working on checkstyle fixes now.

the patch build failure looks like a maven version error

-13

• fixes checkstyle issues noted by test-patch

ugh. okay the compile /javac failures are failing to run the test-compile goal directly, because the dependencies the integration test module needs don't exist until after the package phase.

I think I can fix things so that we don't try to compile the integration tests in that module early. moving out of patch available while I try.

-1 overall

This message was automatically generated.

-1 overall

This message was automatically generated.

We need to fix the precommit errors, but the test failures at least look unrelated.

Sean, anything else we should take care of first? If Avro and HBase work, then IMO we should put this in and iterate in trunk.

-14

• rebased to trunk (4c38f11)
• move compilation of our integration test to after the shaded artifacts they need exist.

-1 overall

This message was automatically generated.

the findbugs failure is YETUS-472

the asflicense failure is YETUS-473

the unit test failure is unrelated.

I think this is ready to go in. There are a few follow-ons I'd like to do afterwards:

• docs providing a quickstart for making and unit testing an application that relies on these artifacts
• making sure nightly runs include the new integration tests
• adding some additional integration tests based on reviewing how some downstream projects make use of minicluster (e.g. hbase, spark)

There are some long-term things I'd like to do, like moving non-IA.Public stuff from the api jar to the runtime jar, but that's going to take a non-trivial amount of effort.

anything else folks can think of?

Above we talked about removing log4j from JobConf to avoid having to add it manually. Is that still planned?

Overall I'm +1 so we can keep iterating in trunk. We can wait until tomorrow before committing to see if anyone else has comments.

yeah, that's a good one!

Committed to trunk, thanks for this awesome undertaking Sean!

As a precursor to proper documentation, could you fill out the release notes field for this JIRA? Also please file follow-on JIRAs as we discussed above.

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11001 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11001/)
HADOOP-11804. Shaded Hadoop client artifacts and minicluster. (wang: rev 70ca1f1e3a328b18eb4e27f7d0f328ae403342d5)

• (add) hadoop-client-modules/hadoop-client-minicluster/pom.xml
• (edit) pom.xml
• (add) hadoop-client-modules/hadoop-client-check-invariants/pom.xml
• (add) hadoop-client-modules/hadoop-client-integration-tests/src/test/resources/hdfs-site.xml
• (add) hadoop-client-modules/hadoop-client-integration-tests/pom.xml
• (add) hadoop-client-modules/hadoop-client-integration-tests/src/test/resources/log4j.properties
• (add) hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/shade/resource/package-info.java
• (delete) hadoop-client/pom.xml
• (edit) hadoop-project/pom.xml
• (add) hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml
• (edit) hadoop-maven-plugins/pom.xml
• (add) hadoop-client-modules/hadoop-client-runtime/pom.xml
• (add) hadoop-client-modules/hadoop-client/pom.xml
• (add) hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java
• (add) hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/shade/resource/ServicesResourceTransformer.java
• (edit) .gitignore
• (add) hadoop-client-modules/hadoop-client-api/pom.xml
• (edit) dev-support/bin/dist-layout-stitching
• (add) hadoop-client-modules/pom.xml
• (edit) hadoop-dist/pom.xml

I think there's a problem with this change. When I change the project version to something that's not already in the Apache maven repository (e.g., 3.0.0-beta1-SNAPSHOT), my build fails:

[ERROR] Failed to execute goal on project hadoop-sls: Could not resolve dependencies for project org.apache.hadoop:hadoop-sls:jar:3.0.0-beta1: Could not find artifact org.apache.hadoop:hadoop-client:jar:3.0.0-beta1 in apache.snapshots.https (https://repository.apache.org/content/repositories/snapshots) -> [Help 1]

It looks like the current trunk is building only because there is a left-over 3.0.0-alpha2-SNAPSHOT of hadoop-client:jar in the repository. The patch changed hadoop-client from jar packaging to pom packaging.

The change to a pom artifact was on purpose. My mistake missing a dependency that relied on the jar version. Andrew Wang, preference between an ADDENDUM patch and a new jira?

Thinking about this more, the change to a pom type instead of an empty jar is better maven hygiene but means that downstream users who don't want to opt-in to the shaded dependencies will have to update their dependency declaration for the change.

i.e.

<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-client</artifactId>
</dependency>

has to change to

<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-client</artifactId>
  <type>pom</type>
</dependency>

Is that what we want? On the positive side, it means downstream folks have to think about wether they want to keep using the client-with-third-party-dependencies or switch to the shaded client. On the negative side, we are forcing all downstream users to think about things instead of just those who are interested in the shaded client.

follow-on JIRAs discussed above now linked as "is related to".

I opened HADOOP-13922 because we didn't have any open bug capturing that the build really is broken. No criticism of Sean Busbey intended.

no worries, I'll move my question over there.

Sean Busbey Thanks for all the hard work. I really appreciate it. I wanted to ask you a related question. I build hadoop all the time and with this patch the amount of time to build trunk has gone up from under 2 mins to 5+ mins on the same hardware. Do you see the same behaviour ? I just wanted to check this was not something on my hardware.

if it is indeed due to shading, is there a way to switch the shading off or have this come into play only when we do a release build. The amount of time spent waiting for a build is a massive hit on productivity.

So any tips on how I can get the old build speed back ?

Thanks
Anu

I assume it would take a new Maven profile to selectively skip shading. Seeing a 2x-3x increase in build times with the 'package' target. Twice, the build just stalled. I think it was stalled at the shading step from the build output.

It would be unwise for us to add a profile to selectively have folks opt-in to producing the shaded artifacts. In Apache HBase that's been a source of build errors where folks end up with empty jars in nexus repos. Here in Hadoop we've been incrementally adding pre-commit coverage for profiles as folks realize they exist and aren't known to pre-commit.

If you don't want to build the shaded artifacts, you can either run just the test goal, or ask maven to only build the module(s) you care about and any modules needed to build those module(s). e.g. mvn --projects :hadoop-yarn-project --also-make package or mvn --projects :hadoop-yarn-ui,:hadoop-yarn-server-timelineservice --also-make package. (You could also ask it to make modules that depend on your changed module with --also-make-dependents but that will almost certainly lead to building the shaded artifacts.)

We could add a profile to selectively opt-out of creating the shaded artifacts, but the above maven commands are how one would do it "the maven way".

A profile to opt-out will be very useful.

Also, is it possible to shade only on package and not on install?

Also, is it possible to shade only on package and not on install?

Not as far as I know. The maven lifecycle specifies that the install goal requires the package goal to have run (ref).

A profile to opt-out will be very useful.

Can one of the three of y'all open a JIRA with the goals specified (i.e. "instructions added to XXXX for developers who want to skip shaded artifact generation") and assign it to me? Once I have a goal location for instructions I'll take on verifying some opt-out mechanism and documentation for same.

I have +1'd the follow-up bug fix patch on HADOOP-13922, but I'm going to hold off committing since it's right before a holiday and others might be offline. I plan to commit it on Monday, 1/2.