Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: build
    • Labels:
      None
    • Target Version/s:
    • Release Note:
      Hide
      <!-- markdown -->

      The `hadoop-client` Maven artifact available in 2.x releases pulls
      Hadoop's transitive dependencies onto a Hadoop application's classpath.
      This can be problematic if the versions of these transitive dependencies
      conflict with the versions used by the application.

      [HADOOP-11804](https://issues.apache.org/jira/browse/HADOOP-11804) adds
      new `hadoop-client-api` and `hadoop-client-runtime` artifacts that
      shade Hadoop's dependencies into a single jar. This avoids leaking
      Hadoop's dependencies onto the application's classpath.
      Show
      <!-- markdown --> The `hadoop-client` Maven artifact available in 2.x releases pulls Hadoop's transitive dependencies onto a Hadoop application's classpath. This can be problematic if the versions of these transitive dependencies conflict with the versions used by the application. [ HADOOP-11804 ]( https://issues.apache.org/jira/browse/HADOOP-11804 ) adds new `hadoop-client-api` and `hadoop-client-runtime` artifacts that shade Hadoop's dependencies into a single jar. This avoids leaking Hadoop's dependencies onto the application's classpath.

      Description

      make a hadoop-client-api and hadoop-client-runtime that i.e. HBase can use to talk with a Hadoop cluster without seeing any of the implementation dependencies.

      see proposal on parent for details.

      1. HADOOP-11804.1.patch
        76 kB
        Sean Busbey
      2. HADOOP-11804.10.patch
        102 kB
        Sean Busbey
      3. HADOOP-11804.11.patch
        121 kB
        Sean Busbey
      4. HADOOP-11804.12.patch
        120 kB
        Sean Busbey
      5. HADOOP-11804.13.patch
        121 kB
        Sean Busbey
      6. HADOOP-11804.14.patch
        122 kB
        Sean Busbey
      7. HADOOP-11804.2.patch
        76 kB
        Sean Busbey
      8. HADOOP-11804.3.patch
        82 kB
        Sean Busbey
      9. HADOOP-11804.4.patch
        86 kB
        Sean Busbey
      10. HADOOP-11804.5.patch
        91 kB
        Sean Busbey
      11. HADOOP-11804.6.patch
        96 kB
        Sean Busbey
      12. HADOOP-11804.7.patch
        139 kB
        Sean Busbey
      13. HADOOP-11804.8.patch
        103 kB
        Sean Busbey
      14. HADOOP-11804.9.patch
        103 kB
        Sean Busbey
      15. hadoop-11804-client-test.tar.gz
        1 kB
        Sean Busbey

        Issue Links

          Activity

          Hide
          busbey Sean Busbey added a comment -

          works-at-all. doesn't filter by IA.Public annotation.

          -01

          • create shaded artifacts for downstream compile, runtime, and test scopes
          • hadoop-client-api is compile scope, currently all org.apache.hadoop from hadoop-client
          • hadoop-client-runtime is runtime scope, contains relocated third party deps from hadoop-client
          • hadoop-client-minicluster is test scope, has the hadoop-minicluster and deps shaded (w/third party relocated) barring those things found in api and runtime.
          • has a local implementation of MSHADE-182
          • requires a maven version that can use the dependency-plugin feature of banning all transitive dependencies.
          Show
          busbey Sean Busbey added a comment - works-at-all. doesn't filter by IA.Public annotation. -01 create shaded artifacts for downstream compile, runtime, and test scopes hadoop-client-api is compile scope, currently all org.apache.hadoop from hadoop-client hadoop-client-runtime is runtime scope, contains relocated third party deps from hadoop-client hadoop-client-minicluster is test scope, has the hadoop-minicluster and deps shaded (w/third party relocated) barring those things found in api and runtime. has a local implementation of MSHADE-182 requires a maven version that can use the dependency-plugin feature of banning all transitive dependencies.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 46s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          -1 javac 7m 39s The patch appears to cause the build to fail.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732376/HADOOP-11804.1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / f24452d
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6674/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 46s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac 7m 39s The patch appears to cause the build to fail. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732376/HADOOP-11804.1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f24452d Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6674/console This message was automatically generated.
          Hide
          busbey Sean Busbey added a comment -

          excellent catch by the QA bot. I had only been doing package / install builds and so didn't notice that enforcer was firing at validate instead of verify.

          -02

          • bind the enforcer rules to the verify phase
          Show
          busbey Sean Busbey added a comment - excellent catch by the QA bot. I had only been doing package / install builds and so didn't notice that enforcer was firing at validate instead of verify. -02 bind the enforcer rules to the verify phase
          Hide
          busbey Sean Busbey added a comment -

          downstream is failing now. need to figure out if I messed something up in my moving commits around branches.

          Show
          busbey Sean Busbey added a comment - downstream is failing now. need to figure out if I messed something up in my moving commits around branches.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 50s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          -1 javac 7m 37s The applied patch generated 1 additional warning messages.
          +1 javadoc 9m 35s There were no new javadoc warning messages.
          -1 release audit 0m 13s The applied patch generated 1 release audit warnings.
          -1 checkstyle 0m 48s The applied patch generated 147 new checkstyle issues (total was 0, now 147).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 2m 27s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          -1 findbugs 1m 3s The patch appears to cause Findbugs (version 2.0.3) to fail.
          +1 native 3m 15s Pre-build of native portion
          -1 client tests 0m 8s Tests failed in hadoop-client.
          +1 maven tests 0m 14s Tests passed in hadoop-maven-plugins.
          +1 hdfs tests 3m 57s Tests passed in bkjournal.
              44m 46s  



          Reason Tests
          Failed build hadoop-client



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732487/HADOOP-11804.2.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / fcd0702
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/6681/artifact/patchprocess/diffJavacWarnings.txt
          Release Audit https://builds.apache.org/job/PreCommit-HADOOP-Build/6681/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6681/artifact/patchprocess/diffcheckstylehadoop-maven-plugins.txt
          hadoop-client test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6681/artifact/patchprocess/testrun_hadoop-client.txt
          hadoop-maven-plugins test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6681/artifact/patchprocess/testrun_hadoop-maven-plugins.txt
          bkjournal test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6681/artifact/patchprocess/testrun_bkjournal.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6681/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6681/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 50s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac 7m 37s The applied patch generated 1 additional warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. -1 release audit 0m 13s The applied patch generated 1 release audit warnings. -1 checkstyle 0m 48s The applied patch generated 147 new checkstyle issues (total was 0, now 147). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 2m 27s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. -1 findbugs 1m 3s The patch appears to cause Findbugs (version 2.0.3) to fail. +1 native 3m 15s Pre-build of native portion -1 client tests 0m 8s Tests failed in hadoop-client. +1 maven tests 0m 14s Tests passed in hadoop-maven-plugins. +1 hdfs tests 3m 57s Tests passed in bkjournal.     44m 46s   Reason Tests Failed build hadoop-client Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732487/HADOOP-11804.2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / fcd0702 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/6681/artifact/patchprocess/diffJavacWarnings.txt Release Audit https://builds.apache.org/job/PreCommit-HADOOP-Build/6681/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6681/artifact/patchprocess/diffcheckstylehadoop-maven-plugins.txt hadoop-client test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6681/artifact/patchprocess/testrun_hadoop-client.txt hadoop-maven-plugins test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6681/artifact/patchprocess/testrun_hadoop-maven-plugins.txt bkjournal test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6681/artifact/patchprocess/testrun_bkjournal.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6681/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6681/console This message was automatically generated.
          Hide
          busbey Sean Busbey added a comment -

          Updated patch, now that HBase master works with Hadoop branch-2 again.

          -03

          • includes a local fork of MSHADE-182 (to handle java services reloction)
          • properly relocate third party references in the client-api bytecode
          • avoid relocating references to JDK packages that are in com.sun
          • avoid shaded jersey binding to javax services

          Known issues right now

          • logging implementations are still relocated
          • HTrace is relocated
            • because of how htrace is implemented, this prevents starting a span in a client app and having the client continue with it as a parent span.
          • some other HBase second-pass tests fail post-rebase that I am still tracking down
            • Notably, all the HBase MR tests fail because the shaded minicluster YARN has some startup problem related to jersey
          Show
          busbey Sean Busbey added a comment - Updated patch, now that HBase master works with Hadoop branch-2 again. -03 includes a local fork of MSHADE-182 (to handle java services reloction) properly relocate third party references in the client-api bytecode avoid relocating references to JDK packages that are in com.sun avoid shaded jersey binding to javax services Known issues right now logging implementations are still relocated HTrace is relocated because of how htrace is implemented, this prevents starting a span in a client app and having the client continue with it as a parent span. some other HBase second-pass tests fail post-rebase that I am still tracking down Notably, all the HBase MR tests fail because the shaded minicluster YARN has some startup problem related to jersey
          Hide
          busbey Sean Busbey added a comment -

          submitting patch just to get a QA look at checkstyle / whitespace. the YARN thing needs to be fixed before this should be pushed.

          Show
          busbey Sean Busbey added a comment - submitting patch just to get a QA look at checkstyle / whitespace. the YARN thing needs to be fixed before this should be pushed.
          Hide
          busbey Sean Busbey added a comment -

          Known issue

          • some of the HBase test failures are because jaxb stuff isn't properly hidden in the shaded minicluster
          Show
          busbey Sean Busbey added a comment - Known issue some of the HBase test failures are because jaxb stuff isn't properly hidden in the shaded minicluster
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 16m 4s Findbugs (version 3.0.0) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          -1 javac 7m 32s The applied patch generated 1 additional warning messages.
          +1 javadoc 9m 40s There were no new javadoc warning messages.
          -1 release audit 0m 13s The applied patch generated 1 release audit warnings.
          -1 checkstyle 0m 48s The applied patch generated 147 new checkstyle issues (total was 0, now 147).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 5m 55s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 findbugs 0m 8s Post-patch findbugs hadoop-client compilation is broken.
          -1 findbugs 0m 16s Post-patch findbugs hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal compilation is broken.
          -1 findbugs 0m 24s Post-patch findbugs hadoop-maven-plugins compilation is broken.
          +1 findbugs 0m 24s The patch does not introduce any new Findbugs (version ) warnings.
          -1 native 0m 8s Failed to build the native portion of hadoop-common prior to running the unit tests in hadoop-client hadoop-maven-plugins hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
              41m 24s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12736190/HADOOP-11804.3.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 7817674
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/6870/artifact/patchprocess/diffJavacWarnings.txt
          Release Audit https://builds.apache.org/job/PreCommit-HADOOP-Build/6870/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6870/artifact/patchprocess/diffcheckstylehadoop-maven-plugins.txt
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6870/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 16m 4s Findbugs (version 3.0.0) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac 7m 32s The applied patch generated 1 additional warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. -1 release audit 0m 13s The applied patch generated 1 release audit warnings. -1 checkstyle 0m 48s The applied patch generated 147 new checkstyle issues (total was 0, now 147). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 5m 55s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 0m 8s Post-patch findbugs hadoop-client compilation is broken. -1 findbugs 0m 16s Post-patch findbugs hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal compilation is broken. -1 findbugs 0m 24s Post-patch findbugs hadoop-maven-plugins compilation is broken. +1 findbugs 0m 24s The patch does not introduce any new Findbugs (version ) warnings. -1 native 0m 8s Failed to build the native portion of hadoop-common prior to running the unit tests in hadoop-client hadoop-maven-plugins hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal     41m 24s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736190/HADOOP-11804.3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 7817674 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/6870/artifact/patchprocess/diffJavacWarnings.txt Release Audit https://builds.apache.org/job/PreCommit-HADOOP-Build/6870/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6870/artifact/patchprocess/diffcheckstylehadoop-maven-plugins.txt Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6870/console This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          This is great work! Looking forward to the final patch. I'm testing it on 2.6 and it mostly works fine. A few issues:

          1. How can an application specify log4j.properties? I tried using both shaded and unshaded class names as appenders and couldn't get it to work.
          2. The Jersey shading is causing problems to WebHDFS. In unit testing my mini DFS cluster cannot start web server. If I use the hadoop-client-runtime jar to talk to a WebHDFS server, the response cannot be correctly parsed. Seems related to ServiceLoader
          Show
          zhz Zhe Zhang added a comment - This is great work! Looking forward to the final patch. I'm testing it on 2.6 and it mostly works fine. A few issues: How can an application specify log4j.properties ? I tried using both shaded and unshaded class names as appenders and couldn't get it to work. The Jersey shading is causing problems to WebHDFS. In unit testing my mini DFS cluster cannot start web server. If I use the hadoop-client-runtime jar to talk to a WebHDFS server, the response cannot be correctly parsed. Seems related to ServiceLoader
          Hide
          busbey Sean Busbey added a comment -

          -04

          • update for current Hadoop 3 branch
          • clean up for dependencies that are already in Java 7+ SE

          limitations:

          • logging libraries still relocated
          • htrace libraries still relocated
          • timeline server excluded from shaded minicluster and marked as optional

          I'm vetting this against HBase now, but figured I'd post an update for some initial review.

          I think the answer for the logging libraries and htrace is to leave them unshaded, since it's common to want to modify logging settings and to want to trace through e.g. the hdfs client. Would like some feedback here.

          Show
          busbey Sean Busbey added a comment - -04 update for current Hadoop 3 branch clean up for dependencies that are already in Java 7+ SE limitations: logging libraries still relocated htrace libraries still relocated timeline server excluded from shaded minicluster and marked as optional I'm vetting this against HBase now, but figured I'd post an update for some initial review. I think the answer for the logging libraries and htrace is to leave them unshaded, since it's common to want to modify logging settings and to want to trace through e.g. the hdfs client. Would like some feedback here.
          Hide
          busbey Sean Busbey added a comment -

          The Jersey shading is causing problems to WebHDFS. In unit testing my mini DFS cluster cannot start web server. If I use the hadoop-client-runtime jar to talk to a WebHDFS server, the response cannot be correctly parsed. Seems related to ServiceLoader

          I have to chase this down still. Zhe Zhang can you help me out with a step-by-step for reproducing?

          Show
          busbey Sean Busbey added a comment - The Jersey shading is causing problems to WebHDFS. In unit testing my mini DFS cluster cannot start web server. If I use the hadoop-client-runtime jar to talk to a WebHDFS server, the response cannot be correctly parsed. Seems related to ServiceLoader I have to chase this down still. Zhe Zhang can you help me out with a step-by-step for reproducing?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 5s HADOOP-11804 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HADOOP-11804
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835636/HADOOP-11804.4.patch
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10928/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s HADOOP-11804 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HADOOP-11804 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835636/HADOOP-11804.4.patch Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10928/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 6s HADOOP-11804 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HADOOP-11804
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835636/HADOOP-11804.4.patch
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10929/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 6s HADOOP-11804 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HADOOP-11804 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835636/HADOOP-11804.4.patch Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10929/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for working on this Sean!

          Not sure why the patch didn't apply for precommit, since it applied for me locally. Since it modifies hadoop-maven-plugins, I "mvn install"'d it first per the normal build instructions.

          Some review comments:

          • Typos: "hte" "htey" "dependnecies" "itis"
          • I see this comment: skip org.apache.avro:avro-ipc because it doesn't look like hadoop-common actually uses it. If there are other issues like this, it'd be nice to surface them in a JIRA comment or JIRA so we can think about fixing them properly.
          • The build failed for me with this error:
          [WARNING] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed with message:
          Duplicate classes found:
          
            Found in:
              org.apache.hadoop:hadoop-client-api:jar:3.0.0-alpha2-SNAPSHOT:compile
              org.apache.hadoop:hadoop-client-minicluster:jar:3.0.0-alpha2-SNAPSHOT:compile
            Duplicate classes:
              org/apache/hadoop/ipc/protobuf/TestRpcServiceProtos$TestProtobufRpc2Proto.class
              org/apache/hadoop/ipc/protobuf/TestProtos$EmptyResponseProto$Builder.class
              org/apache/hadoop/ipc/protobuf/TestProtos$1.class
              org/apache/hadoop/ipc/protobuf/TestProtos$SleepResponseProto$1.class
              org/apache/hadoop/ipc/protobuf/TestRpcServiceProtos$OldProtobufRpcProto$Interface.class
              org/apache/hadoop/ipc/protobuf/TestRpcServiceProtos$NewProtobufRpcProto$BlockingInterface.class
              org/apache/hadoop/ipc/protobuf/TestProtos$AuthMethodResponseProto$1.class
          ...<more proto classes>
          

          trunk is a fast moving target, so if you give me your git hash, I can review against that.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for working on this Sean! Not sure why the patch didn't apply for precommit, since it applied for me locally. Since it modifies hadoop-maven-plugins, I "mvn install"'d it first per the normal build instructions. Some review comments: Typos: "hte" "htey" "dependnecies" "itis" I see this comment: skip org.apache.avro:avro-ipc because it doesn't look like hadoop-common actually uses it . If there are other issues like this, it'd be nice to surface them in a JIRA comment or JIRA so we can think about fixing them properly. The build failed for me with this error: [WARNING] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed with message: Duplicate classes found: Found in: org.apache.hadoop:hadoop-client-api:jar:3.0.0-alpha2-SNAPSHOT:compile org.apache.hadoop:hadoop-client-minicluster:jar:3.0.0-alpha2-SNAPSHOT:compile Duplicate classes: org/apache/hadoop/ipc/protobuf/TestRpcServiceProtos$TestProtobufRpc2Proto.class org/apache/hadoop/ipc/protobuf/TestProtos$EmptyResponseProto$Builder.class org/apache/hadoop/ipc/protobuf/TestProtos$1.class org/apache/hadoop/ipc/protobuf/TestProtos$SleepResponseProto$1.class org/apache/hadoop/ipc/protobuf/TestRpcServiceProtos$OldProtobufRpcProto$Interface.class org/apache/hadoop/ipc/protobuf/TestRpcServiceProtos$NewProtobufRpcProto$BlockingInterface.class org/apache/hadoop/ipc/protobuf/TestProtos$AuthMethodResponseProto$1.class ...<more proto classes> trunk is a fast moving target, so if you give me your git hash, I can review against that.
          Hide
          busbey Sean Busbey added a comment -

          Thanks for the review! I'll update to latest trunk as a part of getting the rest of the feedback addressed.

          I'm a bit concerned about the build failure, since even if you review against my current commit (dbd2057), the hygiene check there (that the same class doesn't end up in more than one of the shaded jars) would still have failed later. I'm confused about test protobuf classes showing up in the client-api jar, but hopefully the cause will be obvious once I rebase.

          I'm planning to un-shade logging and htrace today, so if you'd prefer I not do that please let me know. A related question, do we have a project-wide logging framework?

          Busbey-MBA:hadoop busbey$ git grep -l  "org.slf4j" | grep "\\.java" | wc -l
               320
          Busbey-MBA:hadoop busbey$ git grep -l  "org.apache.commons.logging" | grep "\\.java" | wc -l
              1410
          Busbey-MBA:hadoop busbey$ git grep -l  "java.util.logging" | grep "\\.java" | wc -l
                 2
          Busbey-MBA:hadoop busbey$ git grep -l  "org.apache.log4j" | grep "\\.java" | wc -l
               217
          Busbey-MBA:hadoop busbey$ git grep -l  "org.apache.logging.log4j" | grep "\\.java" | wc -l
                 0
          

          Should I just un-shade all the logging libraries?

          Show
          busbey Sean Busbey added a comment - Thanks for the review! I'll update to latest trunk as a part of getting the rest of the feedback addressed. I'm a bit concerned about the build failure, since even if you review against my current commit (dbd2057), the hygiene check there (that the same class doesn't end up in more than one of the shaded jars) would still have failed later. I'm confused about test protobuf classes showing up in the client-api jar, but hopefully the cause will be obvious once I rebase. I'm planning to un-shade logging and htrace today, so if you'd prefer I not do that please let me know. A related question, do we have a project-wide logging framework? Busbey-MBA:hadoop busbey$ git grep -l "org.slf4j" | grep "\\.java" | wc -l 320 Busbey-MBA:hadoop busbey$ git grep -l "org.apache.commons.logging" | grep "\\.java" | wc -l 1410 Busbey-MBA:hadoop busbey$ git grep -l "java.util.logging" | grep "\\.java" | wc -l 2 Busbey-MBA:hadoop busbey$ git grep -l "org.apache.log4j" | grep "\\.java" | wc -l 217 Busbey-MBA:hadoop busbey$ git grep -l "org.apache.logging.log4j" | grep "\\.java" | wc -l 0 Should I just un-shade all the logging libraries?
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for the quick response Sean,

          I'm confused about test protobuf classes showing up in the client-api jar...

          Yea, this was a mystery to me too. Could be PEBKAC, in which case I'd appreciate fuller build instructions.

          I think the answer for the logging libraries and htrace is to leave them unshaded, since it's common to want to modify logging settings and to want to trace through e.g. the hdfs client. Would like some feedback here.

          The logging rationale here sounds good to me. We've been trying to migrate things to slf4j, but evidently we haven't made much progress.

          Unshading HTrace also sounds good, since I'm guessing that shading will mess up tracing from an app into the Hadoop client. Colin P. McCabe care to comment more authoritatively?

          Show
          andrew.wang Andrew Wang added a comment - Thanks for the quick response Sean, I'm confused about test protobuf classes showing up in the client-api jar... Yea, this was a mystery to me too. Could be PEBKAC, in which case I'd appreciate fuller build instructions. I think the answer for the logging libraries and htrace is to leave them unshaded, since it's common to want to modify logging settings and to want to trace through e.g. the hdfs client. Would like some feedback here. The logging rationale here sounds good to me. We've been trying to migrate things to slf4j, but evidently we haven't made much progress. Unshading HTrace also sounds good, since I'm guessing that shading will mess up tracing from an app into the Hadoop client. Colin P. McCabe care to comment more authoritatively?
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks for the work Sean Busbey! I just did a quick test with the latest patch.

          One high level concern is in terms of maintaining dependencies in the pom's. If a developer adds a new dependency to a module, how would that propagate to these client pom's? Would he/she need to add it to these client pom's for the most part? It wasn't entirely clear to me what that cost of maintenance is. If that is the only way to keep it clean, that's OK. But it would be great if that cost is kept to a minimum.

          1.
          The patch indeed does not apply for me via plain git apply: it breaks with hadoop-client/pom.xml and hadoop-maven-plugins/pom.xml. I did git apply --reject HADOOP-11804.1.patch.

          2.
          Once I fixed the git apply issues, I did mvn clean install package -Pdist -DskipTests -Dmaven.javadoc.skip and it fails right away:

          [ERROR]   The project org.apache.hadoop:hadoop-client-minicluster:3.0.0-alpha2-SNAPSHOT (/Users/sjlee/git/hadoop-trunk/hadoop-client-modules/hadoop-client-minicluster/pom.xml) has 1 error
          [ERROR]     'dependencies.dependency.version' for org.mortbay.jetty:jetty:jar is missing. @ line 266, column 17
          

          I got past it by providing a version for this (chose 6.1.26).

          3.
          The build still fails with a couple of duplicate classes issues. One is what Andrew reported above. Another is duplicate jetty classes.

          [WARNING] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed with message:
          Duplicate classes found:
          
            Found in:
              org.apache.hadoop:hadoop-client-runtime:jar:3.0.0-alpha2-SNAPSHOT:compile
              org.apache.hadoop:hadoop-client-minicluster:jar:3.0.0-alpha2-SNAPSHOT:compile
            Duplicate classes:
              org/apache/hadoop/shaded/org/eclipse/jetty/io/ssl/SslConnection$2.class
              org/apache/hadoop/shaded/org/eclipse/jetty/server/RequestLog.class
              org/apache/hadoop/shaded/org/eclipse/jetty/server/ResourceCache$1.class
              org/apache/hadoop/shaded/org/eclipse/jetty/util/log/AbstractLogger.class
              org/apache/hadoop/shaded/org/eclipse/jetty/util/annotation/Name.class
              org/apache/hadoop/shaded/org/eclipse/jetty/util/component/LifeCycle.class
              org/apache/hadoop/shaded/org/eclipse/jetty/server/HttpChannel$Commit100Callback.class
              org/apache/hadoop/shaded/org/eclipse/jetty/util/ssl/SslContextFactory$1.class
            ...
          

          4.
          Was there a significant difficulty in handing the timeline service v.2? Is it just the number of new dependencies we’re pulling in or the fact that there is a HBase dependency?

          5.
          Regarding the logging libraries, I agree we probably want to exclude them. Things like log4j properties and the way slf4j works can cause issues down the road if shaded.

          Show
          sjlee0 Sangjin Lee added a comment - Thanks for the work Sean Busbey ! I just did a quick test with the latest patch. One high level concern is in terms of maintaining dependencies in the pom's. If a developer adds a new dependency to a module, how would that propagate to these client pom's? Would he/she need to add it to these client pom's for the most part? It wasn't entirely clear to me what that cost of maintenance is. If that is the only way to keep it clean, that's OK. But it would be great if that cost is kept to a minimum. 1. The patch indeed does not apply for me via plain git apply : it breaks with hadoop-client/pom.xml and hadoop-maven-plugins/pom.xml . I did git apply --reject HADOOP-11804 .1.patch . 2. Once I fixed the git apply issues, I did mvn clean install package -Pdist -DskipTests -Dmaven.javadoc.skip and it fails right away: [ERROR] The project org.apache.hadoop:hadoop-client-minicluster:3.0.0-alpha2-SNAPSHOT (/Users/sjlee/git/hadoop-trunk/hadoop-client-modules/hadoop-client-minicluster/pom.xml) has 1 error [ERROR] 'dependencies.dependency.version' for org.mortbay.jetty:jetty:jar is missing. @ line 266, column 17 I got past it by providing a version for this (chose 6.1.26). 3. The build still fails with a couple of duplicate classes issues. One is what Andrew reported above. Another is duplicate jetty classes. [WARNING] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed with message: Duplicate classes found: Found in: org.apache.hadoop:hadoop-client-runtime:jar:3.0.0-alpha2-SNAPSHOT:compile org.apache.hadoop:hadoop-client-minicluster:jar:3.0.0-alpha2-SNAPSHOT:compile Duplicate classes: org/apache/hadoop/shaded/org/eclipse/jetty/io/ssl/SslConnection$2.class org/apache/hadoop/shaded/org/eclipse/jetty/server/RequestLog.class org/apache/hadoop/shaded/org/eclipse/jetty/server/ResourceCache$1.class org/apache/hadoop/shaded/org/eclipse/jetty/util/log/AbstractLogger.class org/apache/hadoop/shaded/org/eclipse/jetty/util/annotation/Name.class org/apache/hadoop/shaded/org/eclipse/jetty/util/component/LifeCycle.class org/apache/hadoop/shaded/org/eclipse/jetty/server/HttpChannel$Commit100Callback.class org/apache/hadoop/shaded/org/eclipse/jetty/util/ssl/SslContextFactory$1.class ... 4. Was there a significant difficulty in handing the timeline service v.2? Is it just the number of new dependencies we’re pulling in or the fact that there is a HBase dependency? 5. Regarding the logging libraries, I agree we probably want to exclude them. Things like log4j properties and the way slf4j works can cause issues down the road if shaded.
          Hide
          busbey Sean Busbey added a comment -

          -05

          • update to current master (0e75496)
          • includes patch for HADOOP-13789 so precommit can run
          • incorporate review feedback
          • exclude slf4j backend from hadoop-client pom
          • leave runtime dependency on htrace, slf4j-api, commons-logging, and log4j

          I made the log4j dependency optional, since there are plenty of use cases where a downstream user wouldn't need it (it's needed for MapReduce and to use some custom Log4j appenders we make).

          Yea, this was a mystery to me too. Could be PEBKAC, in which case I'd appreciate fuller build instructions.

          I don't know if something changed or if I was just building incorrectly before. When I use the build given by Sangjin Lee I easily reproduced this issue. Since it's a general problem (including test classes in two places) I filed HADOOP-13789 to fix it.

          One high level concern is in terms of maintaining dependencies in the pom's. If a developer adds a new dependency to a module, how would that propagate to these client pom's? Would he/she need to add it to these client pom's for the most part? It wasn't entirely clear to me what that cost of maintenance is. If that is the only way to keep it clean, that's OK. But it would be great if that cost is kept to a minimum.

          If a developer adds a new dependency that impacts clients, then the only place they should have to update is the shaded minicluster to make sure the dependency is only included in one of the three shaded artifacts. At the moment, this is the only way to make sure a given class only ends up in one place. Ideally we'd reduce the long term cost by adding a custom shader that can exclude classes that appear in a given dependency. I'm not sure how long creating that shader will take, so I didn't do it yet.

          1. patch doesn't apply
          2. no mortbay.jetty version
          3. duplicate jetty classes

          These were all fallout from the update to jetty 9. They should all be cleaned up in this rebase.

          4. Was there a significant difficulty in handing the timeline service v.2? Is it just the number of new dependencies we’re pulling in or the fact that there is a HBase dependency?

          the volume of dependencies caused me some concern, but HBase getting pulled in by default was the blocker for me. Since I'm trying to test HBase as a downstream application having a shaded hbase dependency included (for a feature that is off by default) seems ill advised.

          Adding it back in for downstream folks that need it should be straight-forward, since it is included as an optional dependency. they just need to add the artifact as a test dependency.

          Show
          busbey Sean Busbey added a comment - -05 update to current master (0e75496) includes patch for HADOOP-13789 so precommit can run incorporate review feedback exclude slf4j backend from hadoop-client pom leave runtime dependency on htrace, slf4j-api, commons-logging, and log4j I made the log4j dependency optional, since there are plenty of use cases where a downstream user wouldn't need it (it's needed for MapReduce and to use some custom Log4j appenders we make). Yea, this was a mystery to me too. Could be PEBKAC, in which case I'd appreciate fuller build instructions. I don't know if something changed or if I was just building incorrectly before. When I use the build given by Sangjin Lee I easily reproduced this issue. Since it's a general problem (including test classes in two places) I filed HADOOP-13789 to fix it. One high level concern is in terms of maintaining dependencies in the pom's. If a developer adds a new dependency to a module, how would that propagate to these client pom's? Would he/she need to add it to these client pom's for the most part? It wasn't entirely clear to me what that cost of maintenance is. If that is the only way to keep it clean, that's OK. But it would be great if that cost is kept to a minimum. If a developer adds a new dependency that impacts clients, then the only place they should have to update is the shaded minicluster to make sure the dependency is only included in one of the three shaded artifacts. At the moment, this is the only way to make sure a given class only ends up in one place. Ideally we'd reduce the long term cost by adding a custom shader that can exclude classes that appear in a given dependency. I'm not sure how long creating that shader will take, so I didn't do it yet. 1. patch doesn't apply 2. no mortbay.jetty version 3. duplicate jetty classes These were all fallout from the update to jetty 9. They should all be cleaned up in this rebase. 4. Was there a significant difficulty in handing the timeline service v.2? Is it just the number of new dependencies we’re pulling in or the fact that there is a HBase dependency? the volume of dependencies caused me some concern, but HBase getting pulled in by default was the blocker for me. Since I'm trying to test HBase as a downstream application having a shaded hbase dependency included (for a feature that is off by default) seems ill advised. Adding it back in for downstream folks that need it should be straight-forward, since it is included as an optional dependency. they just need to add the artifact as a test dependency.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 6m 47s trunk passed
          +1 compile 7m 2s trunk passed
          +1 checkstyle 1m 26s trunk passed
          +1 mvnsite 9m 27s trunk passed
          +1 mvneclipse 1m 4s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client .
          +1 findbugs 1m 44s trunk passed
          +1 javadoc 4m 25s trunk passed
          0 mvndep 0m 23s Maven dependency ordering for patch
          +1 mvninstall 18m 48s the patch passed
          +1 compile 6m 59s the patch passed
          +1 javac 6m 59s the patch passed
          -0 checkstyle 1m 32s root: The patch generated 147 new + 2 unchanged - 0 fixed = 149 total (was 2)
          +1 mvnsite 8m 14s the patch passed
          +1 mvneclipse 1m 13s the patch passed
          -1 whitespace 0m 0s The patch has 7 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 xml 0m 11s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api . hadoop-client-modules hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime
          -1 findbugs 0m 37s hadoop-maven-plugins generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 javadoc 4m 28s the patch passed
          -1 unit 135m 36s root in the patch failed.
          -1 asflicense 0m 30s The patch generated 8 ASF License warnings.
          235m 10s



          Reason Tests
          FindBugs module:hadoop-maven-plugins
            Found reliance on default encoding in org.apache.hadoop.maven.plugin.shade.resource.ServicesResourceTransformer.modifyOutputStream(JarOutputStream):in org.apache.hadoop.maven.plugin.shade.resource.ServicesResourceTransformer.modifyOutputStream(JarOutputStream): new java.io.InputStreamReader(InputStream) At ServicesResourceTransformer.java:[line 117]
            Found reliance on default encoding in org.apache.hadoop.maven.plugin.shade.resource.ServicesResourceTransformer.modifyOutputStream(JarOutputStream):in org.apache.hadoop.maven.plugin.shade.resource.ServicesResourceTransformer.modifyOutputStream(JarOutputStream): new java.io.PrintWriter(OutputStream) At ServicesResourceTransformer.java:[line 116]
          Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-11804
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836761/HADOOP-11804.5.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 0550dc97f3ef 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0e75496
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10974/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10974/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10974/artifact/patchprocess/new-findbugs-hadoop-maven-plugins.html
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10974/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10974/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/10974/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-maven-plugins hadoop-common-project/hadoop-common hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api . hadoop-client-modules hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10974/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 6m 47s trunk passed +1 compile 7m 2s trunk passed +1 checkstyle 1m 26s trunk passed +1 mvnsite 9m 27s trunk passed +1 mvneclipse 1m 4s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client . +1 findbugs 1m 44s trunk passed +1 javadoc 4m 25s trunk passed 0 mvndep 0m 23s Maven dependency ordering for patch +1 mvninstall 18m 48s the patch passed +1 compile 6m 59s the patch passed +1 javac 6m 59s the patch passed -0 checkstyle 1m 32s root: The patch generated 147 new + 2 unchanged - 0 fixed = 149 total (was 2) +1 mvnsite 8m 14s the patch passed +1 mvneclipse 1m 13s the patch passed -1 whitespace 0m 0s The patch has 7 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 11s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api . hadoop-client-modules hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime -1 findbugs 0m 37s hadoop-maven-plugins generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 4m 28s the patch passed -1 unit 135m 36s root in the patch failed. -1 asflicense 0m 30s The patch generated 8 ASF License warnings. 235m 10s Reason Tests FindBugs module:hadoop-maven-plugins   Found reliance on default encoding in org.apache.hadoop.maven.plugin.shade.resource.ServicesResourceTransformer.modifyOutputStream(JarOutputStream):in org.apache.hadoop.maven.plugin.shade.resource.ServicesResourceTransformer.modifyOutputStream(JarOutputStream): new java.io.InputStreamReader(InputStream) At ServicesResourceTransformer.java: [line 117]   Found reliance on default encoding in org.apache.hadoop.maven.plugin.shade.resource.ServicesResourceTransformer.modifyOutputStream(JarOutputStream):in org.apache.hadoop.maven.plugin.shade.resource.ServicesResourceTransformer.modifyOutputStream(JarOutputStream): new java.io.PrintWriter(OutputStream) At ServicesResourceTransformer.java: [line 116] Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-11804 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836761/HADOOP-11804.5.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 0550dc97f3ef 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0e75496 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10974/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10974/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10974/artifact/patchprocess/new-findbugs-hadoop-maven-plugins.html unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10974/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10974/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/10974/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-maven-plugins hadoop-common-project/hadoop-common hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api . hadoop-client-modules hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10974/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          busbey Sean Busbey added a comment -

          updating for precommit feedback now

          Show
          busbey Sean Busbey added a comment - updating for precommit feedback now
          Hide
          andrew.wang Andrew Wang added a comment -
          • It looks like we generate dummy JARs for some of the new modules, since we inherit the jar-plugin executions from the parent pom in hadoop-project-dist. This isn't a big deal, but I have an example for how to disable them at HADOOP-13784.
          • We need to update dev-support/bin/dist-layout-stitching so the new artifacts are included in the release layout. You can run maven with "-Pdist" to test this without running the full create-release script.

          I'm kicking the tires by trying to move Avro to use the shaded api/runtime artifacts. Compile worked, but the MR tests die with this when trying to make a new JobConf:

          testSort(org.apache.avro.mapred.TestWeather)  Time elapsed: 0.011 sec  <<< ERROR!
          java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/org/apache/commons/logging/LogFactory
              at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
                  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
                  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
                  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
                  at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:197)
                  at org.apache.avro.mapred.TestWeather.testSort(TestWeather.java:144)
          

          I looked inside hadoop-client-runtime and don't see commons-logging shaded in there. dependency-tree shows commons-logging being pulled in as a normal dep, which I think is expected.

          I can provide my WIP avro patch if you'd like to play with it yourself, but it was pretty trivial.

          Show
          andrew.wang Andrew Wang added a comment - It looks like we generate dummy JARs for some of the new modules, since we inherit the jar-plugin executions from the parent pom in hadoop-project-dist. This isn't a big deal, but I have an example for how to disable them at HADOOP-13784 . We need to update dev-support/bin/dist-layout-stitching so the new artifacts are included in the release layout. You can run maven with "-Pdist" to test this without running the full create-release script. I'm kicking the tires by trying to move Avro to use the shaded api/runtime artifacts. Compile worked, but the MR tests die with this when trying to make a new JobConf: testSort(org.apache.avro.mapred.TestWeather) Time elapsed: 0.011 sec <<< ERROR! java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/org/apache/commons/logging/LogFactory at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:197) at org.apache.avro.mapred.TestWeather.testSort(TestWeather.java:144) I looked inside hadoop-client-runtime and don't see commons-logging shaded in there. dependency-tree shows commons-logging being pulled in as a normal dep, which I think is expected. I can provide my WIP avro patch if you'd like to play with it yourself, but it was pretty trivial.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Andrew Wang: yes, please do not shade HTrace. We have carefully designed HTrace to not have compatibility issues in 4.x (or with old 3.x code), and shading HTrace will prevent spans from being propagated correctly.

          Show
          cmccabe Colin P. McCabe added a comment - Andrew Wang : yes, please do not shade HTrace. We have carefully designed HTrace to not have compatibility issues in 4.x (or with old 3.x code), and shading HTrace will prevent spans from being propagated correctly.
          Hide
          busbey Sean Busbey added a comment -

          -06

          • rebased to trunk (7154a20)
          • consolidated maven-shade-plugin version
          • updated MSHADE-182 implementation for Hadoop checkstyle rules
          • updated MSHADE-182 implementation for Hadoop's findbugs configs
          • fixed whitespace complaints
          • fixed license header on dependency reduced poms (and cleaned up handling of them generally)

          The failure in TestQueuingContainerManager was consistent, but unrelated to the patch and rebasing has fixed it.

          Show
          busbey Sean Busbey added a comment - -06 rebased to trunk (7154a20) consolidated maven-shade-plugin version updated MSHADE-182 implementation for Hadoop checkstyle rules updated MSHADE-182 implementation for Hadoop's findbugs configs fixed whitespace complaints fixed license header on dependency reduced poms (and cleaned up handling of them generally) The failure in TestQueuingContainerManager was consistent, but unrelated to the patch and rebasing has fixed it.
          Hide
          busbey Sean Busbey added a comment -

          sorry for the noise, v6 is from yesterday afternoon, apparently I forgot to hit submit. I almost have v7 done now with your current feedback Andrew Wang. the class not found exception is a :facepalm: thing I forgot to update in shading.

          Show
          busbey Sean Busbey added a comment - sorry for the noise, v6 is from yesterday afternoon, apparently I forgot to hit submit. I almost have v7 done now with your current feedback Andrew Wang . the class not found exception is a :facepalm: thing I forgot to update in shading.
          Hide
          busbey Sean Busbey added a comment -

          -07

          • rebased to trunk (dbb133c)
          • update to v4 of HADOOP-13789
          • prevent spurious jars
          • make sure dependencies excluded from shading aren't relocated
          • add client jars to dist tarball
          Show
          busbey Sean Busbey added a comment - -07 rebased to trunk (dbb133c) update to v4 of HADOOP-13789 prevent spurious jars make sure dependencies excluded from shading aren't relocated add client jars to dist tarball
          Hide
          andrew.wang Andrew Wang added a comment - - edited

          Thanks for the rev Sean. I tried it with Avro and got NoClassDefFound for Log4J:

          testSort(org.apache.avro.mapred.TestAvroTextSort)  Time elapsed: 0.051 sec  <<< ERROR!
          java.lang.NoClassDefFoundError: org/apache/log4j/Level
          	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
          	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
          	at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:356)
          	at org.apache.avro.mapred.TestAvroTextSort.testSort(TestAvroTextSort.java:37)
          

          I think this is expected based on the contents of the hadoop-client-runtime pom.xml, which marks log4j as optional. I manually added this dependency, and then hit this:

          testReadAvro(org.apache.avro.hadoop.io.TestAvroSequenceFile)  Time elapsed: 0.016 sec  <<< ERROR!
          java.lang.NullPointerException: null
          	at org.apache.hadoop.io.serializer.SerializationFactory.<init>(SerializationFactory.java:58)
          	at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1248)
          	at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1207)
          	at org.apache.avro.hadoop.io.AvroSequenceFile$Writer.<init>(AvroSequenceFile.java:532)
          	at org.apache.avro.hadoop.io.TestAvroSequenceFile.writeSequenceFile(TestAvroSequenceFile.java:200)
          	at org.apache.avro.hadoop.io.TestAvroSequenceFile.testReadAvro(TestAvroSequenceFile.java:53)
          

          I decompiled the SerializationFactory class, and noticed that it messed with the config key. I think we need to add some kind of exclusion for CommonConfigurationKeysPublic.

              // before
              if (conf.get(CommonConfigurationKeys.IO_SERIALIZATIONS_KEY).equals("")) {
              // decompiled
              if (conf.get("org.apache.hadoop.shaded.io.serializations").equals("")) {
          

          Here's my Avro diff for master (without the log4j addition) if you want to try this yourself:

          https://gist.github.com/anonymous/c064c283348a2d1bbec00845678339f9

          Show
          andrew.wang Andrew Wang added a comment - - edited Thanks for the rev Sean. I tried it with Avro and got NoClassDefFound for Log4J: testSort(org.apache.avro.mapred.TestAvroTextSort) Time elapsed: 0.051 sec <<< ERROR! java.lang.NoClassDefFoundError: org/apache/log4j/Level at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:356) at org.apache.avro.mapred.TestAvroTextSort.testSort(TestAvroTextSort.java:37) I think this is expected based on the contents of the hadoop-client-runtime pom.xml, which marks log4j as optional. I manually added this dependency, and then hit this: testReadAvro(org.apache.avro.hadoop.io.TestAvroSequenceFile) Time elapsed: 0.016 sec <<< ERROR! java.lang.NullPointerException: null at org.apache.hadoop.io.serializer.SerializationFactory.<init>(SerializationFactory.java:58) at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1248) at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1207) at org.apache.avro.hadoop.io.AvroSequenceFile$Writer.<init>(AvroSequenceFile.java:532) at org.apache.avro.hadoop.io.TestAvroSequenceFile.writeSequenceFile(TestAvroSequenceFile.java:200) at org.apache.avro.hadoop.io.TestAvroSequenceFile.testReadAvro(TestAvroSequenceFile.java:53) I decompiled the SerializationFactory class, and noticed that it messed with the config key. I think we need to add some kind of exclusion for CommonConfigurationKeysPublic. // before if (conf.get(CommonConfigurationKeys.IO_SERIALIZATIONS_KEY).equals("")) { // decompiled if (conf.get( "org.apache.hadoop.shaded.io.serializations" ).equals("")) { Here's my Avro diff for master (without the log4j addition) if you want to try this yourself: https://gist.github.com/anonymous/c064c283348a2d1bbec00845678339f9
          Hide
          busbey Sean Busbey added a comment -

          I tried it with Avro and got NoClassDefFound for Log4J

          I think this is expected based on the contents of the hadoop-client-runtime pom.xml, which marks log4j as optional.

          Yes, that's correct. Since Avro uses JobConf it will have to add a dependency on log4j. I'll make sure to call that out in docs that provide examples of use. If I wanted to try to refactor out the use of log4j in JobConf, do you think that'd be feasible in time for 3.0?

          I decompiled the SerializationFactory class, and noticed that it messed with the config key. I think we need to add some kind of exclusion for CommonConfigurationKeysPublic.

              // before
              if (conf.get(CommonConfigurationKeys.IO_SERIALIZATIONS_KEY).equals("")) {
              // decompiled
              if (conf.get("org.apache.hadoop.shaded.io.serializations").equals("")) {
          

          Interesting. Let me figure out a test for this and come up with a fix.

          Show
          busbey Sean Busbey added a comment - I tried it with Avro and got NoClassDefFound for Log4J I think this is expected based on the contents of the hadoop-client-runtime pom.xml, which marks log4j as optional. Yes, that's correct. Since Avro uses JobConf it will have to add a dependency on log4j. I'll make sure to call that out in docs that provide examples of use. If I wanted to try to refactor out the use of log4j in JobConf, do you think that'd be feasible in time for 3.0? I decompiled the SerializationFactory class, and noticed that it messed with the config key. I think we need to add some kind of exclusion for CommonConfigurationKeysPublic. // before if (conf.get(CommonConfigurationKeys.IO_SERIALIZATIONS_KEY).equals("")) { // decompiled if (conf.get( "org.apache.hadoop.shaded.io.serializations" ).equals("")) { Interesting. Let me figure out a test for this and come up with a fix.
          Hide
          sjlee0 Sangjin Lee added a comment -

          The javac compiler inlines string or integer primitive constants.

          Show
          sjlee0 Sangjin Lee added a comment - The javac compiler inlines string or integer primitive constants.
          Hide
          busbey Sean Busbey added a comment -

          right, the problem isn't the inlining, it's that the key has been rewritten to use org.apache.hadoop.shaded as a prefix, which presumably none of our actual configuration files use.

          Show
          busbey Sean Busbey added a comment - right, the problem isn't the inlining, it's that the key has been rewritten to use org.apache.hadoop.shaded as a prefix, which presumably none of our actual configuration files use.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Oh got you. I read it too quickly.

          Show
          sjlee0 Sangjin Lee added a comment - Oh got you. I read it too quickly.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 7m 13s trunk passed
          +1 compile 11m 25s trunk passed
          +1 checkstyle 1m 44s trunk passed
          +1 mvnsite 10m 15s trunk passed
          +1 mvneclipse 1m 12s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client hadoop-dist .
          +1 findbugs 13m 33s trunk passed
          +1 javadoc 5m 20s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 29m 27s the patch passed
          +1 compile 9m 36s the patch passed
          +1 javac 9m 36s the patch passed
          +1 checkstyle 1m 49s the patch passed
          -1 mvnsite 1m 43s root in the patch failed.
          +1 mvneclipse 1m 12s the patch passed
          -1 shellcheck 0m 14s The patch generated 76 new + 0 unchanged - 0 fixed = 76 total (was 0)
          -0 shelldocs 0m 8s The patch generated 124 new + 0 unchanged - 0 fixed = 124 total (was 0)
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 24s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project . hadoop-client-modules hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-dist
          -1 findbugs 1m 32s patch/hadoop-common-project/hadoop-common no findbugs output file (hadoop-common-project/hadoop-common/target/findbugsXml.xml)
          -1 findbugs 0m 57s patch/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common no findbugs output file (hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/target/findbugsXml.xml)
          -1 findbugs 0m 40s patch/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle no findbugs output file (hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/target/findbugsXml.xml)
          -1 javadoc 5m 20s root generated 11201 new + 0 unchanged - 0 fixed = 11201 total (was 0)
          -1 unit 9m 34s root in the patch failed.
          -1 asflicense 0m 50s The patch generated 3 ASF License warnings.
          140m 6s



          Reason Tests
          Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.util.TestShell
            hadoop.ha.TestZKFailoverControllerStress
            hadoop.util.TestDataChecksum



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HADOOP-11804
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838047/HADOOP-11804.7.patch
          Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 491e03627500 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / dbb133c
          Default Java 1.8.0_101
          shellcheck v0.4.4
          findbugs v3.0.0
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/artifact/patchprocess/patch-mvnsite-root.txt
          shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/artifact/patchprocess/diff-patch-shellcheck.txt
          shelldocs https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/artifact/patchprocess/diff-patch-shelldocs.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/artifact/patchprocess/patch-findbugs-hadoop-common-project_hadoop-common.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/artifact/patchprocess/patch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-common.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/artifact/patchprocess/patch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/artifact/patchprocess/diff-javadoc-javadoc-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-maven-plugins . hadoop-client-modules hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-common-project/hadoop-common hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 7m 13s trunk passed +1 compile 11m 25s trunk passed +1 checkstyle 1m 44s trunk passed +1 mvnsite 10m 15s trunk passed +1 mvneclipse 1m 12s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client hadoop-dist . +1 findbugs 13m 33s trunk passed +1 javadoc 5m 20s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 29m 27s the patch passed +1 compile 9m 36s the patch passed +1 javac 9m 36s the patch passed +1 checkstyle 1m 49s the patch passed -1 mvnsite 1m 43s root in the patch failed. +1 mvneclipse 1m 12s the patch passed -1 shellcheck 0m 14s The patch generated 76 new + 0 unchanged - 0 fixed = 76 total (was 0) -0 shelldocs 0m 8s The patch generated 124 new + 0 unchanged - 0 fixed = 124 total (was 0) +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 24s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project . hadoop-client-modules hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-dist -1 findbugs 1m 32s patch/hadoop-common-project/hadoop-common no findbugs output file (hadoop-common-project/hadoop-common/target/findbugsXml.xml) -1 findbugs 0m 57s patch/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common no findbugs output file (hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/target/findbugsXml.xml) -1 findbugs 0m 40s patch/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle no findbugs output file (hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/target/findbugsXml.xml) -1 javadoc 5m 20s root generated 11201 new + 0 unchanged - 0 fixed = 11201 total (was 0) -1 unit 9m 34s root in the patch failed. -1 asflicense 0m 50s The patch generated 3 ASF License warnings. 140m 6s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.util.TestShell   hadoop.ha.TestZKFailoverControllerStress   hadoop.util.TestDataChecksum Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HADOOP-11804 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838047/HADOOP-11804.7.patch Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 491e03627500 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dbb133c Default Java 1.8.0_101 shellcheck v0.4.4 findbugs v3.0.0 mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/artifact/patchprocess/patch-mvnsite-root.txt shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/artifact/patchprocess/diff-patch-shellcheck.txt shelldocs https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/artifact/patchprocess/diff-patch-shelldocs.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/artifact/patchprocess/patch-findbugs-hadoop-common-project_hadoop-common.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/artifact/patchprocess/patch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-common.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/artifact/patchprocess/patch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/artifact/patchprocess/diff-javadoc-javadoc-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-maven-plugins . hadoop-client-modules hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-common-project/hadoop-common hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11030/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          If I wanted to try to refactor out the use of log4j in JobConf, do you think that'd be feasible in time for 3.0?

          That'd be awesome! If this is just a mechanical replacement of log4j API usage with SLF4J, someone might pick it up if you file the JIRA.

          Show
          andrew.wang Andrew Wang added a comment - If I wanted to try to refactor out the use of log4j in JobConf, do you think that'd be feasible in time for 3.0? That'd be awesome! If this is just a mechanical replacement of log4j API usage with SLF4J, someone might pick it up if you file the JIRA.
          Hide
          busbey Sean Busbey added a comment -

          odd. the precommit build is marked as aborted and a bunch of the referenced artifacts didn't get archived.

          lemme re-run things.

          Show
          busbey Sean Busbey added a comment - odd. the precommit build is marked as aborted and a bunch of the referenced artifacts didn't get archived. lemme re-run things.
          Hide
          busbey Sean Busbey added a comment -

          If I wanted to try to refactor out the use of log4j in JobConf, do you think that'd be feasible in time for 3.0?

          That'd be awesome! If this is just a mechanical replacement of log4j API usage with SLF4J, someone might pick it up if you file the JIRA.

          I didn't look too closely, but I think it's changing how we internally represent default log levels for jobs. I'll see if I can succinctly phrase the request as a JIRA tomorrow.

          Show
          busbey Sean Busbey added a comment - If I wanted to try to refactor out the use of log4j in JobConf, do you think that'd be feasible in time for 3.0? That'd be awesome! If this is just a mechanical replacement of log4j API usage with SLF4J, someone might pick it up if you file the JIRA. I didn't look too closely, but I think it's changing how we internally represent default log levels for jobs. I'll see if I can succinctly phrase the request as a JIRA tomorrow.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 6s HADOOP-11804 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HADOOP-11804
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838047/HADOOP-11804.7.patch
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11036/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 6s HADOOP-11804 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HADOOP-11804 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838047/HADOOP-11804.7.patch Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11036/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          busbey Sean Busbey added a comment -

          -8

          • rebased to trunk (e1c6ef2)
          • remove patch for HADOOP-13879 now that it's been committed

          Still working on the mis-shading mentioned earlier.

          Show
          busbey Sean Busbey added a comment - -8 rebased to trunk (e1c6ef2) remove patch for HADOOP-13879 now that it's been committed Still working on the mis-shading mentioned earlier.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 1s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 1m 50s Maven dependency ordering for branch
          +1 mvninstall 9m 13s trunk passed
          +1 compile 13m 18s trunk passed
          +1 checkstyle 3m 14s trunk passed
          +1 mvnsite 11m 33s trunk passed
          +1 mvneclipse 1m 21s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client hadoop-dist .
          +1 findbugs 15m 21s trunk passed
          +1 javadoc 4m 25s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 29m 33s the patch passed
          +1 compile 11m 19s the patch passed
          +1 javac 11m 19s the patch passed
          +1 checkstyle 1m 56s root: The patch generated 0 new + 0 unchanged - 2 fixed = 0 total (was 2)
          +1 mvnsite 11m 13s the patch passed
          +1 mvneclipse 2m 3s the patch passed
          +1 shellcheck 0m 14s There were no new shellcheck issues.
          +1 shelldocs 0m 9s There were no new shelldocs issues.
          -1 whitespace 0m 1s The patch 3 line(s) with tabs.
          +1 xml 0m 34s The patch has no ill-formed XML file.
          0 findbugs 0m 1s Skipped patched modules with no Java source: hadoop-project . hadoop-client-modules hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-dist
          -1 findbugs 0m 27s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml)
          +1 javadoc 5m 33s the patch passed
          -1 unit 115m 15s root in the patch failed.
          -1 asflicense 1m 26s The patch generated 5 ASF License warnings.
          286m 13s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.fs.viewfs.TestViewFsHdfs
            hadoop.hdfs.server.datanode.TestDirectoryScanner
            hadoop.hdfs.server.namenode.ha.TestHASafeMode



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HADOOP-11804
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838047/HADOOP-11804.7.patch
          Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 6d3657dc2602 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 29e3b34
          Default Java 1.8.0_101
          shellcheck v0.4.4
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11035/artifact/patchprocess/whitespace-tabs.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11035/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11035/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11035/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11035/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-maven-plugins . hadoop-client-modules hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-common-project/hadoop-common hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11035/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 1s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 1m 50s Maven dependency ordering for branch +1 mvninstall 9m 13s trunk passed +1 compile 13m 18s trunk passed +1 checkstyle 3m 14s trunk passed +1 mvnsite 11m 33s trunk passed +1 mvneclipse 1m 21s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client hadoop-dist . +1 findbugs 15m 21s trunk passed +1 javadoc 4m 25s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 29m 33s the patch passed +1 compile 11m 19s the patch passed +1 javac 11m 19s the patch passed +1 checkstyle 1m 56s root: The patch generated 0 new + 0 unchanged - 2 fixed = 0 total (was 2) +1 mvnsite 11m 13s the patch passed +1 mvneclipse 2m 3s the patch passed +1 shellcheck 0m 14s There were no new shellcheck issues. +1 shelldocs 0m 9s There were no new shelldocs issues. -1 whitespace 0m 1s The patch 3 line(s) with tabs. +1 xml 0m 34s The patch has no ill-formed XML file. 0 findbugs 0m 1s Skipped patched modules with no Java source: hadoop-project . hadoop-client-modules hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-dist -1 findbugs 0m 27s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml) +1 javadoc 5m 33s the patch passed -1 unit 115m 15s root in the patch failed. -1 asflicense 1m 26s The patch generated 5 ASF License warnings. 286m 13s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.fs.viewfs.TestViewFsHdfs   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.namenode.ha.TestHASafeMode Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HADOOP-11804 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838047/HADOOP-11804.7.patch Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 6d3657dc2602 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 29e3b34 Default Java 1.8.0_101 shellcheck v0.4.4 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11035/artifact/patchprocess/whitespace-tabs.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11035/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11035/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11035/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11035/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-maven-plugins . hadoop-client-modules hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-common-project/hadoop-common hadoop-dist hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11035/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          busbey Sean Busbey added a comment -

          11035 is from another aborted build before the timeout change.

          Show
          busbey Sean Busbey added a comment - 11035 is from another aborted build before the timeout change.
          Hide
          busbey Sean Busbey added a comment -

          -08

          • rebase to trunk (aab9737)
          • don't relocate the Hadoop config keys that start with "io." or "net."

          I'm going to work through the Avro example tomorrow morning, just want to get this put up for precommit and visibility in the mean time.

          I couldn't get an automated test easily. We essentially need an enforcer rule that can get the contents of String constants in a set of class files. I included a comment in the enforce-invariants pom of how I checked this manually. could use some better formatting. Not sure if it's worth pushing this check into dev-support and then invoking ant-run in the verify phase to run it or something like that.

          Show
          busbey Sean Busbey added a comment - -08 rebase to trunk (aab9737) don't relocate the Hadoop config keys that start with "io." or "net." I'm going to work through the Avro example tomorrow morning, just want to get this put up for precommit and visibility in the mean time. I couldn't get an automated test easily. We essentially need an enforcer rule that can get the contents of String constants in a set of class files. I included a comment in the enforce-invariants pom of how I checked this manually. could use some better formatting. Not sure if it's worth pushing this check into dev-support and then invoking ant-run in the verify phase to run it or something like that.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 6m 52s trunk passed
          +1 compile 9m 39s trunk passed
          +1 checkstyle 1m 36s trunk passed
          +1 mvnsite 9m 45s trunk passed
          +1 mvneclipse 1m 8s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client hadoop-dist .
          +1 findbugs 0m 30s trunk passed
          +1 javadoc 4m 28s trunk passed
          0 mvndep 0m 22s Maven dependency ordering for patch
          +1 mvninstall 25m 40s the patch passed
          +1 compile 9m 25s the patch passed
          +1 javac 9m 25s the patch passed
          +1 checkstyle 1m 41s the patch passed
          +1 mvnsite 8m 26s the patch passed
          +1 mvneclipse 1m 17s the patch passed
          +1 shellcheck 0m 12s There were no new shellcheck issues.
          +1 shelldocs 0m 9s There were no new shelldocs issues.
          -1 whitespace 0m 0s The patch 3 line(s) with tabs.
          +1 xml 0m 11s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-invariants hadoop-dist hadoop-client-modules .
          +1 findbugs 0m 42s the patch passed
          +1 javadoc 4m 34s the patch passed
          -1 unit 100m 42s root in the patch failed.
          -1 asflicense 0m 47s The patch generated 5 ASF License warnings.
          211m 35s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-11804
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839324/HADOOP-11804.8.patch
          Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 0756b372143f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / aab9737
          Default Java 1.8.0_111
          shellcheck v0.4.5
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11086/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11086/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11086/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11086/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-maven-plugins hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-invariants hadoop-dist hadoop-client-modules . U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11086/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 6m 52s trunk passed +1 compile 9m 39s trunk passed +1 checkstyle 1m 36s trunk passed +1 mvnsite 9m 45s trunk passed +1 mvneclipse 1m 8s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client hadoop-dist . +1 findbugs 0m 30s trunk passed +1 javadoc 4m 28s trunk passed 0 mvndep 0m 22s Maven dependency ordering for patch +1 mvninstall 25m 40s the patch passed +1 compile 9m 25s the patch passed +1 javac 9m 25s the patch passed +1 checkstyle 1m 41s the patch passed +1 mvnsite 8m 26s the patch passed +1 mvneclipse 1m 17s the patch passed +1 shellcheck 0m 12s There were no new shellcheck issues. +1 shelldocs 0m 9s There were no new shelldocs issues. -1 whitespace 0m 0s The patch 3 line(s) with tabs. +1 xml 0m 11s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-invariants hadoop-dist hadoop-client-modules . +1 findbugs 0m 42s the patch passed +1 javadoc 4m 34s the patch passed -1 unit 100m 42s root in the patch failed. -1 asflicense 0m 47s The patch generated 5 ASF License warnings. 211m 35s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-11804 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839324/HADOOP-11804.8.patch Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 0756b372143f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aab9737 Default Java 1.8.0_111 shellcheck v0.4.5 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11086/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11086/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11086/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11086/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-maven-plugins hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-invariants hadoop-dist hadoop-client-modules . U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11086/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          busbey Sean Busbey added a comment -
          Lines that start with ????? in the ASF License  report indicate files that do not have an Apache license header:
           !????? /testptch/hadoop/hadoop-build-tools/maven-eclipse.xml
           !????? /testptch/hadoop/hadoop-build-tools/.externalToolBuilders/Maven_Ant_Builder.launch
           !????? hadoop-client/.classpath
           !????? hadoop-client/.project
           !????? hadoop-client/.settings/org.eclipse.jdt.core.prefs
          

          this looks like a gap between our building and yetus. Anyone have an idea of how I get those temporary files to show up? I don't see any of them locally.

          In any case, none of these are files from this patch (and the hadoop-client directory shouldn't exist after the patch).

          Show
          busbey Sean Busbey added a comment - Lines that start with ????? in the ASF License report indicate files that do not have an Apache license header: !????? /testptch/hadoop/hadoop-build-tools/maven-eclipse.xml !????? /testptch/hadoop/hadoop-build-tools/.externalToolBuilders/Maven_Ant_Builder.launch !????? hadoop-client/.classpath !????? hadoop-client/.project !????? hadoop-client/.settings/org.eclipse.jdt.core.prefs this looks like a gap between our building and yetus. Anyone have an idea of how I get those temporary files to show up? I don't see any of them locally. In any case, none of these are files from this patch (and the hadoop-client directory shouldn't exist after the patch).
          Hide
          busbey Sean Busbey added a comment -

          okay, mvn eclipse:eclipse makes them show up. I'm pretty sure we don't run that as a part of the release processs, so these files shouldn't end up published. we should add them to the ignore list for the rat-plugin, but that's not related to this ticket specifically.

          Show
          busbey Sean Busbey added a comment - okay, mvn eclipse:eclipse makes them show up. I'm pretty sure we don't run that as a part of the release processs, so these files shouldn't end up published. we should add them to the ignore list for the rat-plugin, but that's not related to this ticket specifically.
          Hide
          busbey Sean Busbey added a comment -

          -9

          • rebased to trunk (b2d4b7b)
          • fixed tabs

          now back to working on avro.

          Show
          busbey Sean Busbey added a comment - -9 rebased to trunk (b2d4b7b) fixed tabs now back to working on avro.
          Hide
          busbey Sean Busbey added a comment -

          Okay, I've updated Andrew Wang's patch for avro master: https://gist.github.com/busbey/caccd2fedda36409df4a319deb3e69c2

          And given v9 locally installed, I get a clean run of ./build.sh test in lang/java.

          the avro-mapred module ends up with this dep tree:

          [INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ avro-mapred ---
          [INFO] org.apache.avro:avro-mapred:bundle:1.9.0-SNAPSHOT
          [INFO] +- org.apache.avro:avro-ipc:jar:1.9.0-SNAPSHOT:compile
          [INFO] |  +- org.apache.avro:avro:jar:1.9.0-SNAPSHOT:compile
          [INFO] |  +- org.mortbay.jetty:jetty:jar:6.1.26:compile
          [INFO] |  +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
          [INFO] |  +- io.netty:netty:jar:3.5.13.Final:compile
          [INFO] |  +- org.apache.velocity:velocity:jar:1.7:compile
          [INFO] |  |  +- commons-collections:commons-collections:jar:3.2.1:compile
          [INFO] |  |  \- commons-lang:commons-lang:jar:2.4:compile
          [INFO] |  \- org.mortbay.jetty:servlet-api:jar:2.5-20081211:compile
          [INFO] +- org.apache.avro:avro-ipc:jar:tests:1.9.0-SNAPSHOT:test
          [INFO] +- org.apache.avro:avro:test-jar:tests:1.9.0-SNAPSHOT:test
          [INFO] |  +- org.apache.avro:avro-guava-dependencies:jar:1.9.0-SNAPSHOT:compile
          [INFO] |  |  +- com.google.guava:guava:jar:11.0.2:compile
          [INFO] |  |  \- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
          [INFO] |  +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
          [INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.1.3:compile
          [INFO] |  +- org.apache.commons:commons-compress:jar:1.8.1:compile
          [INFO] |  \- org.tukaani:xz:jar:1.5:compile
          [INFO] +- org.easymock:easymock:jar:3.2:test
          [INFO] |  +- cglib:cglib-nodep:jar:2.2.2:test
          [INFO] |  \- org.objenesis:objenesis:jar:1.3:test
          [INFO] +- org.hamcrest:hamcrest-library:jar:1.3:test
          [INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
          [INFO] +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
          [INFO] +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
          [INFO] +- commons-codec:commons-codec:jar:1.9:compile
          [INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.0.0-alpha2-SNAPSHOT:compile
          [INFO] +- org.apache.hadoop:hadoop-client-runtime:jar:3.0.0-alpha2-SNAPSHOT:provided
          [INFO] |  +- org.apache.htrace:htrace-core4:jar:4.1.0-incubating:provided
          [INFO] |  \- commons-logging:commons-logging:jar:1.1.3:provided
          [INFO] +- log4j:log4j:jar:1.2.17:provided
          [INFO] +- junit:junit:jar:4.11:test
          [INFO] +- org.slf4j:slf4j-api:jar:1.7.7:compile
          [INFO] \- org.slf4j:slf4j-simple:jar:1.7.7:runtime
          

          Which looks about as I'd expect (maybe hadoop-client-api should be provided as well, given the notes in the avro pom, but that's probably immaterial).

          and avro-tools looks reasonable:

          [INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ avro-tools ---
          [INFO] org.apache.avro:avro-tools:jar:1.9.0-SNAPSHOT
          [INFO] +- org.apache.avro:avro:jar:1.9.0-SNAPSHOT:compile
          [INFO] |  +- org.apache.avro:avro-guava-dependencies:jar:1.9.0-SNAPSHOT:compile
          [INFO] |  |  \- com.google.guava:guava:jar:11.0.2:compile
          [INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
          [INFO] |  +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
          [INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.1.3:compile
          [INFO] |  +- org.apache.commons:commons-compress:jar:1.8.1:compile
          [INFO] |  \- org.tukaani:xz:jar:1.5:compile
          [INFO] +- org.apache.avro:avro:jar:tests:1.9.0-SNAPSHOT:test
          [INFO] +- org.apache.avro:avro-compiler:jar:1.9.0-SNAPSHOT:compile
          [INFO] |  +- commons-lang:commons-lang:jar:2.6:compile
          [INFO] |  +- org.apache.velocity:velocity:jar:1.7:compile
          [INFO] |  |  \- commons-collections:commons-collections:jar:3.2.1:compile
          [INFO] |  \- joda-time:joda-time:jar:2.7:compile
          [INFO] +- org.apache.avro:avro-ipc:jar:1.9.0-SNAPSHOT:compile
          [INFO] |  +- org.mortbay.jetty:jetty:jar:6.1.26:compile
          [INFO] |  +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
          [INFO] |  +- io.netty:netty:jar:3.5.13.Final:compile
          [INFO] |  \- org.mortbay.jetty:servlet-api:jar:2.5-20081211:compile
          [INFO] +- org.apache.avro:avro-mapred:jar:1.9.0-SNAPSHOT:compile
          [INFO] |  \- commons-codec:commons-codec:jar:1.9:compile
          [INFO] +- org.apache.avro:avro-mapred:test-jar:tests:1.9.0-SNAPSHOT:test
          [INFO] +- commons-cli:commons-cli:jar:1.2:compile
          [INFO] +- commons-logging:commons-logging:jar:1.1.1:compile
          [INFO] +- commons-httpclient:commons-httpclient:jar:3.1:compile
          [INFO] +- org.apache.avro:trevni-core:jar:1.9.0-SNAPSHOT:compile
          [INFO] +- org.apache.avro:trevni-avro:jar:1.9.0-SNAPSHOT:compile
          [INFO] |  \- org.apache.avro:avro-mapred:jar:hadoop2:1.9.0-SNAPSHOT:compile
          [INFO] +- org.apache.avro:trevni-core:jar:tests:1.9.0-SNAPSHOT:compile
          [INFO] +- org.apache.avro:trevni-avro:jar:tests:1.9.0-SNAPSHOT:compile
          [INFO] +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
          [INFO] +- org.apache.hadoop:hadoop-client-runtime:jar:3.0.0-alpha2-SNAPSHOT:runtime
          [INFO] |  \- org.apache.htrace:htrace-core4:jar:4.1.0-incubating:runtime
          [INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.0.0-alpha2-SNAPSHOT:compile
          [INFO] +- log4j:log4j:jar:1.2.17:compile
          [INFO] +- net.sf.jopt-simple:jopt-simple:jar:4.7:compile
          [INFO] +- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
          [INFO] +- junit:junit:jar:4.11:test
          [INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
          [INFO] +- org.slf4j:slf4j-api:jar:1.7.7:compile
          [INFO] \- org.slf4j:slf4j-simple:jar:1.7.7:runtime
          
          Show
          busbey Sean Busbey added a comment - Okay, I've updated Andrew Wang 's patch for avro master: https://gist.github.com/busbey/caccd2fedda36409df4a319deb3e69c2 And given v9 locally installed, I get a clean run of ./build.sh test in lang/java . the avro-mapred module ends up with this dep tree: [INFO] --- maven-dependency-plugin:2.8:tree ( default -cli) @ avro-mapred --- [INFO] org.apache.avro:avro-mapred:bundle:1.9.0-SNAPSHOT [INFO] +- org.apache.avro:avro-ipc:jar:1.9.0-SNAPSHOT:compile [INFO] | +- org.apache.avro:avro:jar:1.9.0-SNAPSHOT:compile [INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:compile [INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile [INFO] | +- io.netty:netty:jar:3.5.13.Final:compile [INFO] | +- org.apache.velocity:velocity:jar:1.7:compile [INFO] | | +- commons-collections:commons-collections:jar:3.2.1:compile [INFO] | | \- commons-lang:commons-lang:jar:2.4:compile [INFO] | \- org.mortbay.jetty:servlet-api:jar:2.5-20081211:compile [INFO] +- org.apache.avro:avro-ipc:jar:tests:1.9.0-SNAPSHOT:test [INFO] +- org.apache.avro:avro:test-jar:tests:1.9.0-SNAPSHOT:test [INFO] | +- org.apache.avro:avro-guava-dependencies:jar:1.9.0-SNAPSHOT:compile [INFO] | | +- com.google.guava:guava:jar:11.0.2:compile [INFO] | | \- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile [INFO] | +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile [INFO] | +- org.xerial.snappy:snappy-java:jar:1.1.1.3:compile [INFO] | +- org.apache.commons:commons-compress:jar:1.8.1:compile [INFO] | \- org.tukaani:xz:jar:1.5:compile [INFO] +- org.easymock:easymock:jar:3.2:test [INFO] | +- cglib:cglib-nodep:jar:2.2.2:test [INFO] | \- org.objenesis:objenesis:jar:1.3:test [INFO] +- org.hamcrest:hamcrest-library:jar:1.3:test [INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test [INFO] +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile [INFO] +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile [INFO] +- commons-codec:commons-codec:jar:1.9:compile [INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.0.0-alpha2-SNAPSHOT:compile [INFO] +- org.apache.hadoop:hadoop-client-runtime:jar:3.0.0-alpha2-SNAPSHOT:provided [INFO] | +- org.apache.htrace:htrace-core4:jar:4.1.0-incubating:provided [INFO] | \- commons-logging:commons-logging:jar:1.1.3:provided [INFO] +- log4j:log4j:jar:1.2.17:provided [INFO] +- junit:junit:jar:4.11:test [INFO] +- org.slf4j:slf4j-api:jar:1.7.7:compile [INFO] \- org.slf4j:slf4j-simple:jar:1.7.7:runtime Which looks about as I'd expect (maybe hadoop-client-api should be provided as well, given the notes in the avro pom, but that's probably immaterial). and avro-tools looks reasonable: [INFO] --- maven-dependency-plugin:2.8:tree ( default -cli) @ avro-tools --- [INFO] org.apache.avro:avro-tools:jar:1.9.0-SNAPSHOT [INFO] +- org.apache.avro:avro:jar:1.9.0-SNAPSHOT:compile [INFO] | +- org.apache.avro:avro-guava-dependencies:jar:1.9.0-SNAPSHOT:compile [INFO] | | \- com.google.guava:guava:jar:11.0.2:compile [INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile [INFO] | +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile [INFO] | +- org.xerial.snappy:snappy-java:jar:1.1.1.3:compile [INFO] | +- org.apache.commons:commons-compress:jar:1.8.1:compile [INFO] | \- org.tukaani:xz:jar:1.5:compile [INFO] +- org.apache.avro:avro:jar:tests:1.9.0-SNAPSHOT:test [INFO] +- org.apache.avro:avro-compiler:jar:1.9.0-SNAPSHOT:compile [INFO] | +- commons-lang:commons-lang:jar:2.6:compile [INFO] | +- org.apache.velocity:velocity:jar:1.7:compile [INFO] | | \- commons-collections:commons-collections:jar:3.2.1:compile [INFO] | \- joda-time:joda-time:jar:2.7:compile [INFO] +- org.apache.avro:avro-ipc:jar:1.9.0-SNAPSHOT:compile [INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:compile [INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile [INFO] | +- io.netty:netty:jar:3.5.13.Final:compile [INFO] | \- org.mortbay.jetty:servlet-api:jar:2.5-20081211:compile [INFO] +- org.apache.avro:avro-mapred:jar:1.9.0-SNAPSHOT:compile [INFO] | \- commons-codec:commons-codec:jar:1.9:compile [INFO] +- org.apache.avro:avro-mapred:test-jar:tests:1.9.0-SNAPSHOT:test [INFO] +- commons-cli:commons-cli:jar:1.2:compile [INFO] +- commons-logging:commons-logging:jar:1.1.1:compile [INFO] +- commons-httpclient:commons-httpclient:jar:3.1:compile [INFO] +- org.apache.avro:trevni-core:jar:1.9.0-SNAPSHOT:compile [INFO] +- org.apache.avro:trevni-avro:jar:1.9.0-SNAPSHOT:compile [INFO] | \- org.apache.avro:avro-mapred:jar:hadoop2:1.9.0-SNAPSHOT:compile [INFO] +- org.apache.avro:trevni-core:jar:tests:1.9.0-SNAPSHOT:compile [INFO] +- org.apache.avro:trevni-avro:jar:tests:1.9.0-SNAPSHOT:compile [INFO] +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile [INFO] +- org.apache.hadoop:hadoop-client-runtime:jar:3.0.0-alpha2-SNAPSHOT:runtime [INFO] | \- org.apache.htrace:htrace-core4:jar:4.1.0-incubating:runtime [INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.0.0-alpha2-SNAPSHOT:compile [INFO] +- log4j:log4j:jar:1.2.17:compile [INFO] +- net.sf.jopt-simple:jopt-simple:jar:4.7:compile [INFO] +- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile [INFO] +- junit:junit:jar:4.11:test [INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test [INFO] +- org.slf4j:slf4j-api:jar:1.7.7:compile [INFO] \- org.slf4j:slf4j-simple:jar:1.7.7:runtime
          Hide
          aw Allen Wittenauer added a comment -

          Do we know yet what the impact is going to be on the hadoop-tools components? Especially with the "on demand" jar detection at build time?

          Show
          aw Allen Wittenauer added a comment - Do we know yet what the impact is going to be on the hadoop-tools components? Especially with the "on demand" jar detection at build time?
          Hide
          busbey Sean Busbey added a comment -

          This shouldn't impact the hadoop-tools components AFAICT, since the shaded client jars are opt-in. Only one of the hadoop-tools components uses the hadoop-client dependency (which still works), the others directly reference more specific internal-facing dependencies.

          Do you have an example of the kind of problem you're expecting?

          Show
          busbey Sean Busbey added a comment - This shouldn't impact the hadoop-tools components AFAICT, since the shaded client jars are opt-in. Only one of the hadoop-tools components uses the hadoop-client dependency (which still works), the others directly reference more specific internal-facing dependencies. Do you have an example of the kind of problem you're expecting?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 6m 51s trunk passed
          +1 compile 9m 37s trunk passed
          +1 checkstyle 1m 38s trunk passed
          +1 mvnsite 9m 38s trunk passed
          +1 mvneclipse 1m 8s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client hadoop-dist .
          +1 findbugs 0m 29s trunk passed
          +1 javadoc 4m 27s trunk passed
          0 mvndep 0m 23s Maven dependency ordering for patch
          +1 mvninstall 26m 29s the patch passed
          +1 compile 9m 35s the patch passed
          +1 javac 9m 35s the patch passed
          +1 checkstyle 1m 42s the patch passed
          +1 mvnsite 8m 25s the patch passed
          +1 mvneclipse 1m 19s the patch passed
          +1 shellcheck 0m 12s There were no new shellcheck issues.
          +1 shelldocs 0m 9s There were no new shelldocs issues.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 13s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-invariants hadoop-dist hadoop-client-modules .
          +1 findbugs 0m 46s the patch passed
          +1 javadoc 5m 13s the patch passed
          -1 unit 102m 2s root in the patch failed.
          -1 asflicense 0m 46s The patch generated 5 ASF License warnings.
          214m 16s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-11804
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839405/HADOOP-11804.9.patch
          Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 369e09964d55 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b2d4b7b
          Default Java 1.8.0_111
          shellcheck v0.4.5
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11092/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11092/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11092/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-maven-plugins hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-invariants hadoop-dist hadoop-client-modules . U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11092/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 6m 51s trunk passed +1 compile 9m 37s trunk passed +1 checkstyle 1m 38s trunk passed +1 mvnsite 9m 38s trunk passed +1 mvneclipse 1m 8s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client hadoop-dist . +1 findbugs 0m 29s trunk passed +1 javadoc 4m 27s trunk passed 0 mvndep 0m 23s Maven dependency ordering for patch +1 mvninstall 26m 29s the patch passed +1 compile 9m 35s the patch passed +1 javac 9m 35s the patch passed +1 checkstyle 1m 42s the patch passed +1 mvnsite 8m 25s the patch passed +1 mvneclipse 1m 19s the patch passed +1 shellcheck 0m 12s There were no new shellcheck issues. +1 shelldocs 0m 9s There were no new shelldocs issues. +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 13s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-invariants hadoop-dist hadoop-client-modules . +1 findbugs 0m 46s the patch passed +1 javadoc 5m 13s the patch passed -1 unit 102m 2s root in the patch failed. -1 asflicense 0m 46s The patch generated 5 ASF License warnings. 214m 16s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-11804 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839405/HADOOP-11804.9.patch Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 369e09964d55 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b2d4b7b Default Java 1.8.0_111 shellcheck v0.4.5 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11092/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11092/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11092/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-maven-plugins hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-invariants hadoop-dist hadoop-client-modules . U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11092/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          busbey Sean Busbey added a comment -

          Any feedback Sangjin Lee?

          Zhe Zhang any more details here:

          The Jersey shading is causing problems to WebHDFS. In unit testing my mini DFS cluster cannot start web server. If I use the hadoop-client-runtime jar to talk to a WebHDFS server, the response cannot be correctly parsed. Seems related to ServiceLoader

          I have to chase this down still. Zhe Zhang can you help me out with a step-by-step for reproducing?

          Show
          busbey Sean Busbey added a comment - Any feedback Sangjin Lee ? Zhe Zhang any more details here: The Jersey shading is causing problems to WebHDFS. In unit testing my mini DFS cluster cannot start web server. If I use the hadoop-client-runtime jar to talk to a WebHDFS server, the response cannot be correctly parsed. Seems related to ServiceLoader I have to chase this down still. Zhe Zhang can you help me out with a step-by-step for reproducing?
          Hide
          sjlee0 Sangjin Lee added a comment -

          I'll take a look at it soon. It would be great if there is a reproducible case that we can easily replicate the problem.

          If it is related with the ServiceLoader, you want to look at the properties files from which the ServiceLoader loads classes via reflection (e.g. META-INF/services/org.apache.hadoop.fs.FileSystem). The class names in these files need to match the actual class names in the jar. If the actual class was shaded, the content of these properties files need to be shaded too.

          Show
          sjlee0 Sangjin Lee added a comment - I'll take a look at it soon. It would be great if there is a reproducible case that we can easily replicate the problem. If it is related with the ServiceLoader, you want to look at the properties files from which the ServiceLoader loads classes via reflection (e.g. META-INF/services/org.apache.hadoop.fs.FileSystem). The class names in these files need to match the actual class names in the jar. If the actual class was shaded, the content of these properties files need to be shaded too.
          Hide
          busbey Sean Busbey added a comment -
          • hadoop-11804-client-test.tar.gz

          Including an example program and the steps I used to verify that it can use shaded client/runtime to talk to a WebHDFS instance.

          Show
          busbey Sean Busbey added a comment - hadoop-11804-client-test.tar.gz Including an example program and the steps I used to verify that it can use shaded client/runtime to talk to a WebHDFS instance.
          Hide
          busbey Sean Busbey added a comment -

          -10

          • rebased for trunk (0cfd7ad)
          • cleaned up new pom uses of duplicate group ID

          ping Sangjin Lee, Zhe Zhang, Andrew Wang.

          I think the attached example shows v10 working against a webHDFS instance. Please let me know if there's a different test folks would prefer.

          Show
          busbey Sean Busbey added a comment - -10 rebased for trunk (0cfd7ad) cleaned up new pom uses of duplicate group ID ping Sangjin Lee , Zhe Zhang , Andrew Wang . I think the attached example shows v10 working against a webHDFS instance. Please let me know if there's a different test folks would prefer.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 1m 41s Maven dependency ordering for branch
          +1 mvninstall 7m 41s trunk passed
          +1 compile 11m 20s trunk passed
          +1 checkstyle 1m 46s trunk passed
          +1 mvnsite 11m 15s trunk passed
          +1 mvneclipse 1m 28s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client hadoop-dist .
          +1 findbugs 0m 32s trunk passed
          +1 javadoc 4m 52s trunk passed
          0 mvndep 0m 24s Maven dependency ordering for patch
          +1 mvninstall 27m 30s the patch passed
          +1 compile 11m 33s the patch passed
          +1 javac 11m 33s the patch passed
          +1 checkstyle 2m 0s the patch passed
          +1 mvnsite 9m 17s the patch passed
          +1 mvneclipse 1m 27s the patch passed
          +1 shellcheck 0m 13s There were no new shellcheck issues.
          +1 shelldocs 0m 10s There were no new shelldocs issues.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 13s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api . hadoop-client-modules hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-dist
          +1 findbugs 0m 52s the patch passed
          +1 javadoc 5m 26s the patch passed
          -1 unit 117m 53s root in the patch failed.
          -1 asflicense 0m 50s The patch generated 5 ASF License warnings.
          241m 54s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestDecommissionWithStriped
            hadoop.hdfs.TestCrcCorruption
            hadoop.yarn.server.timeline.webapp.TestTimelineWebServices



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-11804
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841606/HADOOP-11804.10.patch
          Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 3a2ef8c2e283 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f885160
          Default Java 1.8.0_111
          shellcheck v0.4.5
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11191/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11191/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11191/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-maven-plugins hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api . hadoop-client-modules hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11191/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 1m 41s Maven dependency ordering for branch +1 mvninstall 7m 41s trunk passed +1 compile 11m 20s trunk passed +1 checkstyle 1m 46s trunk passed +1 mvnsite 11m 15s trunk passed +1 mvneclipse 1m 28s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client hadoop-dist . +1 findbugs 0m 32s trunk passed +1 javadoc 4m 52s trunk passed 0 mvndep 0m 24s Maven dependency ordering for patch +1 mvninstall 27m 30s the patch passed +1 compile 11m 33s the patch passed +1 javac 11m 33s the patch passed +1 checkstyle 2m 0s the patch passed +1 mvnsite 9m 17s the patch passed +1 mvneclipse 1m 27s the patch passed +1 shellcheck 0m 13s There were no new shellcheck issues. +1 shelldocs 0m 10s There were no new shelldocs issues. +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 13s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api . hadoop-client-modules hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-dist +1 findbugs 0m 52s the patch passed +1 javadoc 5m 26s the patch passed -1 unit 117m 53s root in the patch failed. -1 asflicense 0m 50s The patch generated 5 ASF License warnings. 241m 54s Reason Tests Failed junit tests hadoop.hdfs.TestDecommissionWithStriped   hadoop.hdfs.TestCrcCorruption   hadoop.yarn.server.timeline.webapp.TestTimelineWebServices Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-11804 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841606/HADOOP-11804.10.patch Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 3a2ef8c2e283 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f885160 Default Java 1.8.0_111 shellcheck v0.4.5 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11191/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11191/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11191/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-maven-plugins hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api . hadoop-client-modules hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-dist U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11191/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          busbey Sean Busbey added a comment -

          this weekend I found an issue while setting up some integration tests. should have an update later today.

          Show
          busbey Sean Busbey added a comment - this weekend I found an issue while setting up some integration tests. should have an update later today.
          Hide
          busbey Sean Busbey added a comment -

          -11

          • adds an integration test for using the shaded minicluster
          • fixes some found gaps in dependencies for runtime v minicluster
          • fixes a typo in the relocation of javax.servlet for minicluster

          known limitation: the integration test I added for webhdfs access currently fails. I believe this is the failure Zhe Zhang was originally referring to. I'm still trying to figure out a workable solution.

          Show
          busbey Sean Busbey added a comment - -11 adds an integration test for using the shaded minicluster fixes some found gaps in dependencies for runtime v minicluster fixes a typo in the relocation of javax.servlet for minicluster known limitation: the integration test I added for webhdfs access currently fails. I believe this is the failure Zhe Zhang was originally referring to. I'm still trying to figure out a workable solution.
          Hide
          busbey Sean Busbey added a comment -

          -12

          • rebase to trunk (c265515)
          • updates shaded minicluster to add in the java services declarations we excluded from shaded runtime
          • integration test for webhdfs now passes

          A limitation of the Java Services API (and that some of the services we have implementations for are part of Java servlets) is that folks using the shaded minicluster for testing will see our shaded implementations for a few APIs.

          I believe as of this patch Zhe Zhang's original problem should be solved. The previously attached example should also provide a manual way of verifying running against a live cluster.

          Show
          busbey Sean Busbey added a comment - -12 rebase to trunk (c265515) updates shaded minicluster to add in the java services declarations we excluded from shaded runtime integration test for webhdfs now passes A limitation of the Java Services API (and that some of the services we have implementations for are part of Java servlets) is that folks using the shaded minicluster for testing will see our shaded implementations for a few APIs. I believe as of this patch Zhe Zhang 's original problem should be solved. The previously attached example should also provide a manual way of verifying running against a live cluster.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 1m 46s Maven dependency ordering for branch
          +1 mvninstall 7m 43s trunk passed
          +1 compile 10m 8s trunk passed
          +1 checkstyle 1m 47s trunk passed
          +1 mvnsite 11m 23s trunk passed
          +1 mvneclipse 1m 15s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client hadoop-dist .
          +1 findbugs 0m 29s trunk passed
          +1 javadoc 4m 27s trunk passed
          0 mvndep 0m 23s Maven dependency ordering for patch
          +1 mvninstall 27m 48s the patch passed
          -1 compile 9m 25s root in the patch failed.
          -1 javac 9m 25s root in the patch failed.
          -0 checkstyle 1m 43s root: The patch generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0)
          -1 mvnsite 8m 28s root in the patch failed.
          +1 mvneclipse 1m 20s the patch passed
          +1 shellcheck 0m 12s There were no new shellcheck issues.
          +1 shelldocs 0m 9s There were no new shelldocs issues.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 16s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api . hadoop-client-modules hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-check-test-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-dist
          -1 findbugs 0m 23s patch/hadoop-client-modules/hadoop-client-integration-tests no findbugs output file (hadoop-client-modules/hadoop-client-integration-tests/target/findbugsXml.xml)
          +1 javadoc 4m 33s the patch passed
          -1 unit 103m 45s root in the patch failed.
          -1 asflicense 0m 47s The patch generated 5 ASF License warnings.
          221m 58s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker
            hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
            hadoop.yarn.server.timeline.webapp.TestTimelineWebServices



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-11804
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842359/HADOOP-11804.11.patch
          Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux b0305f744376 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0ef7961
          Default Java 1.8.0_111
          shellcheck v0.4.5
          findbugs v3.0.0
          compile https://builds.apache.org/job/PreCommit-HADOOP-Build/11216/artifact/patchprocess/patch-compile-root.txt
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11216/artifact/patchprocess/patch-compile-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11216/artifact/patchprocess/diff-checkstyle-root.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/11216/artifact/patchprocess/patch-mvnsite-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11216/artifact/patchprocess/patch-findbugs-hadoop-client-modules_hadoop-client-integration-tests.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11216/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11216/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11216/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-maven-plugins hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api . hadoop-client-modules hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-check-test-invariants hadoop-client-modules/hadoop-client-integration-tests hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11216/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 1m 46s Maven dependency ordering for branch +1 mvninstall 7m 43s trunk passed +1 compile 10m 8s trunk passed +1 checkstyle 1m 47s trunk passed +1 mvnsite 11m 23s trunk passed +1 mvneclipse 1m 15s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client hadoop-dist . +1 findbugs 0m 29s trunk passed +1 javadoc 4m 27s trunk passed 0 mvndep 0m 23s Maven dependency ordering for patch +1 mvninstall 27m 48s the patch passed -1 compile 9m 25s root in the patch failed. -1 javac 9m 25s root in the patch failed. -0 checkstyle 1m 43s root: The patch generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) -1 mvnsite 8m 28s root in the patch failed. +1 mvneclipse 1m 20s the patch passed +1 shellcheck 0m 12s There were no new shellcheck issues. +1 shelldocs 0m 9s There were no new shelldocs issues. +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 16s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api . hadoop-client-modules hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-check-test-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-dist -1 findbugs 0m 23s patch/hadoop-client-modules/hadoop-client-integration-tests no findbugs output file (hadoop-client-modules/hadoop-client-integration-tests/target/findbugsXml.xml) +1 javadoc 4m 33s the patch passed -1 unit 103m 45s root in the patch failed. -1 asflicense 0m 47s The patch generated 5 ASF License warnings. 221m 58s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker   hadoop.hdfs.server.namenode.TestNamenodeCapacityReport   hadoop.yarn.server.timeline.webapp.TestTimelineWebServices Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-11804 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842359/HADOOP-11804.11.patch Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux b0305f744376 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0ef7961 Default Java 1.8.0_111 shellcheck v0.4.5 findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-HADOOP-Build/11216/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11216/artifact/patchprocess/patch-compile-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11216/artifact/patchprocess/diff-checkstyle-root.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/11216/artifact/patchprocess/patch-mvnsite-root.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11216/artifact/patchprocess/patch-findbugs-hadoop-client-modules_hadoop-client-integration-tests.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11216/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11216/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11216/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-maven-plugins hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api . hadoop-client-modules hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-check-test-invariants hadoop-client-modules/hadoop-client-integration-tests hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-dist U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11216/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          busbey Sean Busbey added a comment -

          findbugs is a false negative caused by a test-file-only module. working on checkstyle fixes now.

          the patch build failure looks like a maven version error

          Show
          busbey Sean Busbey added a comment - findbugs is a false negative caused by a test-file-only module. working on checkstyle fixes now. the patch build failure looks like a maven version error
          Hide
          busbey Sean Busbey added a comment -

          -13

          • fixes checkstyle issues noted by test-patch
          Show
          busbey Sean Busbey added a comment - -13 fixes checkstyle issues noted by test-patch
          Hide
          busbey Sean Busbey added a comment -

          ugh. okay the compile /javac failures are failing to run the test-compile goal directly, because the dependencies the integration test module needs don't exist until after the package phase.

          Show
          busbey Sean Busbey added a comment - ugh. okay the compile /javac failures are failing to run the test-compile goal directly, because the dependencies the integration test module needs don't exist until after the package phase.
          Hide
          busbey Sean Busbey added a comment -

          I think I can fix things so that we don't try to compile the integration tests in that module early. moving out of patch available while I try.

          Show
          busbey Sean Busbey added a comment - I think I can fix things so that we don't try to compile the integration tests in that module early. moving out of patch available while I try.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 1m 46s Maven dependency ordering for branch
          +1 mvninstall 7m 20s trunk passed
          +1 compile 11m 14s trunk passed
          +1 checkstyle 1m 46s trunk passed
          +1 mvnsite 10m 59s trunk passed
          +1 mvneclipse 1m 11s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client hadoop-dist .
          +1 findbugs 0m 29s trunk passed
          +1 javadoc 4m 40s trunk passed
          0 mvndep 0m 24s Maven dependency ordering for patch
          +1 mvninstall 28m 24s the patch passed
          -1 compile 9m 25s root in the patch failed.
          -1 javac 9m 25s root in the patch failed.
          -0 checkstyle 1m 43s root: The patch generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0)
          -1 mvnsite 8m 29s root in the patch failed.
          +1 mvneclipse 1m 21s the patch passed
          +1 shellcheck 0m 11s There were no new shellcheck issues.
          +1 shelldocs 0m 9s There were no new shelldocs issues.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 15s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api . hadoop-client-modules hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-check-test-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-dist
          -1 findbugs 0m 23s patch/hadoop-client-modules/hadoop-client-integration-tests no findbugs output file (hadoop-client-modules/hadoop-client-integration-tests/target/findbugsXml.xml)
          +1 javadoc 4m 31s the patch passed
          -1 unit 108m 12s root in the patch failed.
          -1 asflicense 0m 49s The patch generated 5 ASF License warnings.
          227m 36s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
            hadoop.yarn.server.nodemanager.webapp.TestNMWebServices
            hadoop.yarn.server.timeline.webapp.TestTimelineWebServices



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-11804
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842394/HADOOP-11804.12.patch
          Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 3f72b140438c 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c265515
          Default Java 1.8.0_111
          shellcheck v0.4.5
          findbugs v3.0.0
          compile https://builds.apache.org/job/PreCommit-HADOOP-Build/11219/artifact/patchprocess/patch-compile-root.txt
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11219/artifact/patchprocess/patch-compile-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11219/artifact/patchprocess/diff-checkstyle-root.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/11219/artifact/patchprocess/patch-mvnsite-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11219/artifact/patchprocess/patch-findbugs-hadoop-client-modules_hadoop-client-integration-tests.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11219/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11219/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11219/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-maven-plugins hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api . hadoop-client-modules hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-check-test-invariants hadoop-client-modules/hadoop-client-integration-tests hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11219/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 1m 46s Maven dependency ordering for branch +1 mvninstall 7m 20s trunk passed +1 compile 11m 14s trunk passed +1 checkstyle 1m 46s trunk passed +1 mvnsite 10m 59s trunk passed +1 mvneclipse 1m 11s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client hadoop-dist . +1 findbugs 0m 29s trunk passed +1 javadoc 4m 40s trunk passed 0 mvndep 0m 24s Maven dependency ordering for patch +1 mvninstall 28m 24s the patch passed -1 compile 9m 25s root in the patch failed. -1 javac 9m 25s root in the patch failed. -0 checkstyle 1m 43s root: The patch generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0) -1 mvnsite 8m 29s root in the patch failed. +1 mvneclipse 1m 21s the patch passed +1 shellcheck 0m 11s There were no new shellcheck issues. +1 shelldocs 0m 9s There were no new shelldocs issues. +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 15s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api . hadoop-client-modules hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-check-test-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-dist -1 findbugs 0m 23s patch/hadoop-client-modules/hadoop-client-integration-tests no findbugs output file (hadoop-client-modules/hadoop-client-integration-tests/target/findbugsXml.xml) +1 javadoc 4m 31s the patch passed -1 unit 108m 12s root in the patch failed. -1 asflicense 0m 49s The patch generated 5 ASF License warnings. 227m 36s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery   hadoop.yarn.server.nodemanager.webapp.TestNMWebServices   hadoop.yarn.server.timeline.webapp.TestTimelineWebServices Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-11804 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842394/HADOOP-11804.12.patch Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 3f72b140438c 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c265515 Default Java 1.8.0_111 shellcheck v0.4.5 findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-HADOOP-Build/11219/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11219/artifact/patchprocess/patch-compile-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11219/artifact/patchprocess/diff-checkstyle-root.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/11219/artifact/patchprocess/patch-mvnsite-root.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11219/artifact/patchprocess/patch-findbugs-hadoop-client-modules_hadoop-client-integration-tests.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11219/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11219/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11219/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-maven-plugins hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api . hadoop-client-modules hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-check-test-invariants hadoop-client-modules/hadoop-client-integration-tests hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-dist U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11219/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 7m 15s trunk passed
          +1 compile 10m 1s trunk passed
          +1 checkstyle 1m 37s trunk passed
          +1 mvnsite 9m 45s trunk passed
          +1 mvneclipse 1m 9s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client hadoop-dist .
          +1 findbugs 0m 29s trunk passed
          +1 javadoc 4m 28s trunk passed
          0 mvndep 0m 23s Maven dependency ordering for patch
          +1 mvninstall 27m 21s the patch passed
          -1 compile 10m 44s root in the patch failed.
          -1 javac 10m 44s root in the patch failed.
          +1 checkstyle 1m 53s the patch passed
          -1 mvnsite 9m 44s root in the patch failed.
          +1 mvneclipse 1m 31s the patch passed
          +1 shellcheck 0m 12s There were no new shellcheck issues.
          +1 shelldocs 0m 10s There were no new shelldocs issues.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 16s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants . hadoop-client-modules hadoop-dist
          -1 findbugs 0m 23s patch/hadoop-client-modules/hadoop-client-integration-tests no findbugs output file (hadoop-client-modules/hadoop-client-integration-tests/target/findbugsXml.xml)
          +1 javadoc 4m 33s the patch passed
          -1 unit 109m 10s root in the patch failed.
          -1 asflicense 0m 46s The patch generated 5 ASF License warnings.
          225m 40s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestFsDatasetCache
            hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
            hadoop.yarn.server.nodemanager.webapp.TestNMWebServices



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-11804
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842401/HADOOP-11804.13.patch
          Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 623fd25b745f 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c265515
          Default Java 1.8.0_111
          shellcheck v0.4.5
          findbugs v3.0.0
          compile https://builds.apache.org/job/PreCommit-HADOOP-Build/11220/artifact/patchprocess/patch-compile-root.txt
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11220/artifact/patchprocess/patch-compile-root.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/11220/artifact/patchprocess/patch-mvnsite-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11220/artifact/patchprocess/patch-findbugs-hadoop-client-modules_hadoop-client-integration-tests.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11220/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11220/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11220/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-maven-plugins hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants hadoop-client-modules/hadoop-client-integration-tests . hadoop-client-modules hadoop-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11220/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 7m 15s trunk passed +1 compile 10m 1s trunk passed +1 checkstyle 1m 37s trunk passed +1 mvnsite 9m 45s trunk passed +1 mvneclipse 1m 9s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client hadoop-dist . +1 findbugs 0m 29s trunk passed +1 javadoc 4m 28s trunk passed 0 mvndep 0m 23s Maven dependency ordering for patch +1 mvninstall 27m 21s the patch passed -1 compile 10m 44s root in the patch failed. -1 javac 10m 44s root in the patch failed. +1 checkstyle 1m 53s the patch passed -1 mvnsite 9m 44s root in the patch failed. +1 mvneclipse 1m 31s the patch passed +1 shellcheck 0m 12s There were no new shellcheck issues. +1 shelldocs 0m 10s There were no new shelldocs issues. +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 16s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants . hadoop-client-modules hadoop-dist -1 findbugs 0m 23s patch/hadoop-client-modules/hadoop-client-integration-tests no findbugs output file (hadoop-client-modules/hadoop-client-integration-tests/target/findbugsXml.xml) +1 javadoc 4m 33s the patch passed -1 unit 109m 10s root in the patch failed. -1 asflicense 0m 46s The patch generated 5 ASF License warnings. 225m 40s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestFsDatasetCache   hadoop.yarn.server.timeline.webapp.TestTimelineWebServices   hadoop.yarn.server.nodemanager.webapp.TestNMWebServices Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-11804 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842401/HADOOP-11804.13.patch Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 623fd25b745f 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c265515 Default Java 1.8.0_111 shellcheck v0.4.5 findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-HADOOP-Build/11220/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/11220/artifact/patchprocess/patch-compile-root.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/11220/artifact/patchprocess/patch-mvnsite-root.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11220/artifact/patchprocess/patch-findbugs-hadoop-client-modules_hadoop-client-integration-tests.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11220/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11220/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11220/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-maven-plugins hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants hadoop-client-modules/hadoop-client-integration-tests . hadoop-client-modules hadoop-dist U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11220/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          We need to fix the precommit errors, but the test failures at least look unrelated.

          Sean, anything else we should take care of first? If Avro and HBase work, then IMO we should put this in and iterate in trunk.

          Show
          andrew.wang Andrew Wang added a comment - We need to fix the precommit errors, but the test failures at least look unrelated. Sean, anything else we should take care of first? If Avro and HBase work, then IMO we should put this in and iterate in trunk.
          Hide
          busbey Sean Busbey added a comment - - edited

          -14

          • rebased to trunk (4c38f11)
          • move compilation of our integration test to after the shaded artifacts they need exist.
          Show
          busbey Sean Busbey added a comment - - edited -14 rebased to trunk (4c38f11) move compilation of our integration test to after the shaded artifacts they need exist.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 1m 22s Maven dependency ordering for branch
          +1 mvninstall 6m 50s trunk passed
          +1 compile 9m 37s trunk passed
          +1 checkstyle 1m 36s trunk passed
          +1 mvnsite 9m 44s trunk passed
          +1 mvneclipse 1m 10s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client hadoop-dist .
          +1 findbugs 0m 29s trunk passed
          +1 javadoc 4m 28s trunk passed
          0 mvndep 0m 24s Maven dependency ordering for patch
          +1 mvninstall 27m 32s the patch passed
          +1 compile 9m 24s the patch passed
          +1 javac 9m 24s the patch passed
          +1 checkstyle 1m 41s the patch passed
          +1 mvnsite 8m 24s the patch passed
          +1 mvneclipse 1m 20s the patch passed
          +1 shellcheck 0m 11s There were no new shellcheck issues.
          +1 shelldocs 0m 9s There were no new shelldocs issues.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 15s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants . hadoop-client-modules hadoop-dist
          -1 findbugs 0m 23s patch/hadoop-client-modules/hadoop-client-integration-tests no findbugs output file (hadoop-client-modules/hadoop-client-integration-tests/target/findbugsXml.xml)
          +1 javadoc 4m 35s the patch passed
          -1 unit 13m 59s root in the patch failed.
          -1 asflicense 0m 44s The patch generated 5 ASF License warnings.
          128m 3s



          Reason Tests
          Failed junit tests hadoop.security.token.delegation.web.TestWebDelegationToken



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-11804
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842651/HADOOP-11804.14.patch
          Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 10335af6950a 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4c38f11
          Default Java 1.8.0_111
          shellcheck v0.4.5
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11239/artifact/patchprocess/patch-findbugs-hadoop-client-modules_hadoop-client-integration-tests.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11239/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11239/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11239/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-maven-plugins hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants hadoop-client-modules/hadoop-client-integration-tests . hadoop-client-modules hadoop-dist U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11239/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 1m 22s Maven dependency ordering for branch +1 mvninstall 6m 50s trunk passed +1 compile 9m 37s trunk passed +1 checkstyle 1m 36s trunk passed +1 mvnsite 9m 44s trunk passed +1 mvneclipse 1m 10s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client hadoop-dist . +1 findbugs 0m 29s trunk passed +1 javadoc 4m 28s trunk passed 0 mvndep 0m 24s Maven dependency ordering for patch +1 mvninstall 27m 32s the patch passed +1 compile 9m 24s the patch passed +1 javac 9m 24s the patch passed +1 checkstyle 1m 41s the patch passed +1 mvnsite 8m 24s the patch passed +1 mvneclipse 1m 20s the patch passed +1 shellcheck 0m 11s There were no new shellcheck issues. +1 shelldocs 0m 9s There were no new shelldocs issues. +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 15s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants . hadoop-client-modules hadoop-dist -1 findbugs 0m 23s patch/hadoop-client-modules/hadoop-client-integration-tests no findbugs output file (hadoop-client-modules/hadoop-client-integration-tests/target/findbugsXml.xml) +1 javadoc 4m 35s the patch passed -1 unit 13m 59s root in the patch failed. -1 asflicense 0m 44s The patch generated 5 ASF License warnings. 128m 3s Reason Tests Failed junit tests hadoop.security.token.delegation.web.TestWebDelegationToken Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-11804 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842651/HADOOP-11804.14.patch Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 10335af6950a 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4c38f11 Default Java 1.8.0_111 shellcheck v0.4.5 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11239/artifact/patchprocess/patch-findbugs-hadoop-client-modules_hadoop-client-integration-tests.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11239/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11239/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11239/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-maven-plugins hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants hadoop-client-modules/hadoop-client-integration-tests . hadoop-client-modules hadoop-dist U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11239/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          busbey Sean Busbey added a comment -

          the findbugs failure is YETUS-472

          the asflicense failure is YETUS-473

          the unit test failure is unrelated.

          Show
          busbey Sean Busbey added a comment - the findbugs failure is YETUS-472 the asflicense failure is YETUS-473 the unit test failure is unrelated.
          Hide
          busbey Sean Busbey added a comment -

          I think this is ready to go in. There are a few follow-ons I'd like to do afterwards:

          • docs providing a quickstart for making and unit testing an application that relies on these artifacts
          • making sure nightly runs include the new integration tests
          • adding some additional integration tests based on reviewing how some downstream projects make use of minicluster (e.g. hbase, spark)

          There are some long-term things I'd like to do, like moving non-IA.Public stuff from the api jar to the runtime jar, but that's going to take a non-trivial amount of effort.

          anything else folks can think of?

          Show
          busbey Sean Busbey added a comment - I think this is ready to go in. There are a few follow-ons I'd like to do afterwards: docs providing a quickstart for making and unit testing an application that relies on these artifacts making sure nightly runs include the new integration tests adding some additional integration tests based on reviewing how some downstream projects make use of minicluster (e.g. hbase, spark) There are some long-term things I'd like to do, like moving non-IA.Public stuff from the api jar to the runtime jar, but that's going to take a non-trivial amount of effort. anything else folks can think of?
          Hide
          andrew.wang Andrew Wang added a comment -

          Above we talked about removing log4j from JobConf to avoid having to add it manually. Is that still planned?

          Overall I'm +1 so we can keep iterating in trunk. We can wait until tomorrow before committing to see if anyone else has comments.

          Show
          andrew.wang Andrew Wang added a comment - Above we talked about removing log4j from JobConf to avoid having to add it manually. Is that still planned? Overall I'm +1 so we can keep iterating in trunk. We can wait until tomorrow before committing to see if anyone else has comments.
          Hide
          busbey Sean Busbey added a comment -

          yeah, that's a good one!

          Show
          busbey Sean Busbey added a comment - yeah, that's a good one!
          Hide
          andrew.wang Andrew Wang added a comment -

          Committed to trunk, thanks for this awesome undertaking Sean!

          As a precursor to proper documentation, could you fill out the release notes field for this JIRA? Also please file follow-on JIRAs as we discussed above.

          Show
          andrew.wang Andrew Wang added a comment - Committed to trunk, thanks for this awesome undertaking Sean! As a precursor to proper documentation, could you fill out the release notes field for this JIRA? Also please file follow-on JIRAs as we discussed above.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11001 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11001/)
          HADOOP-11804. Shaded Hadoop client artifacts and minicluster. (wang: rev 70ca1f1e3a328b18eb4e27f7d0f328ae403342d5)

          • (add) hadoop-client-modules/hadoop-client-minicluster/pom.xml
          • (edit) pom.xml
          • (add) hadoop-client-modules/hadoop-client-check-invariants/pom.xml
          • (add) hadoop-client-modules/hadoop-client-integration-tests/src/test/resources/hdfs-site.xml
          • (add) hadoop-client-modules/hadoop-client-integration-tests/pom.xml
          • (add) hadoop-client-modules/hadoop-client-integration-tests/src/test/resources/log4j.properties
          • (add) hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/shade/resource/package-info.java
          • (delete) hadoop-client/pom.xml
          • (edit) hadoop-project/pom.xml
          • (add) hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml
          • (edit) hadoop-maven-plugins/pom.xml
          • (add) hadoop-client-modules/hadoop-client-runtime/pom.xml
          • (add) hadoop-client-modules/hadoop-client/pom.xml
          • (add) hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java
          • (add) hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/shade/resource/ServicesResourceTransformer.java
          • (edit) .gitignore
          • (add) hadoop-client-modules/hadoop-client-api/pom.xml
          • (edit) dev-support/bin/dist-layout-stitching
          • (add) hadoop-client-modules/pom.xml
          • (edit) hadoop-dist/pom.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11001 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11001/ ) HADOOP-11804 . Shaded Hadoop client artifacts and minicluster. (wang: rev 70ca1f1e3a328b18eb4e27f7d0f328ae403342d5) (add) hadoop-client-modules/hadoop-client-minicluster/pom.xml (edit) pom.xml (add) hadoop-client-modules/hadoop-client-check-invariants/pom.xml (add) hadoop-client-modules/hadoop-client-integration-tests/src/test/resources/hdfs-site.xml (add) hadoop-client-modules/hadoop-client-integration-tests/pom.xml (add) hadoop-client-modules/hadoop-client-integration-tests/src/test/resources/log4j.properties (add) hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/shade/resource/package-info.java (delete) hadoop-client/pom.xml (edit) hadoop-project/pom.xml (add) hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml (edit) hadoop-maven-plugins/pom.xml (add) hadoop-client-modules/hadoop-client-runtime/pom.xml (add) hadoop-client-modules/hadoop-client/pom.xml (add) hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java (add) hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/shade/resource/ServicesResourceTransformer.java (edit) .gitignore (add) hadoop-client-modules/hadoop-client-api/pom.xml (edit) dev-support/bin/dist-layout-stitching (add) hadoop-client-modules/pom.xml (edit) hadoop-dist/pom.xml
          Hide
          jpallas Joe Pallas added a comment -

          I think there's a problem with this change. When I change the project version to something that's not already in the Apache maven repository (e.g., 3.0.0-beta1-SNAPSHOT), my build fails:

          [ERROR] Failed to execute goal on project hadoop-sls: Could not resolve dependencies for project org.apache.hadoop:hadoop-sls:jar:3.0.0-beta1: Could not find artifact org.apache.hadoop:hadoop-client:jar:3.0.0-beta1 in apache.snapshots.https (https://repository.apache.org/content/repositories/snapshots) -> [Help 1]
          

          It looks like the current trunk is building only because there is a left-over 3.0.0-alpha2-SNAPSHOT of hadoop-client:jar in the repository. The patch changed hadoop-client from jar packaging to pom packaging.

          Show
          jpallas Joe Pallas added a comment - I think there's a problem with this change. When I change the project version to something that's not already in the Apache maven repository (e.g., 3.0.0-beta1-SNAPSHOT), my build fails: [ERROR] Failed to execute goal on project hadoop-sls: Could not resolve dependencies for project org.apache.hadoop:hadoop-sls:jar:3.0.0-beta1: Could not find artifact org.apache.hadoop:hadoop-client:jar:3.0.0-beta1 in apache.snapshots.https (https://repository.apache.org/content/repositories/snapshots) -> [Help 1] It looks like the current trunk is building only because there is a left-over 3.0.0-alpha2-SNAPSHOT of hadoop-client:jar in the repository. The patch changed hadoop-client from jar packaging to pom packaging.
          Hide
          busbey Sean Busbey added a comment -

          The change to a pom artifact was on purpose. My mistake missing a dependency that relied on the jar version. Andrew Wang, preference between an ADDENDUM patch and a new jira?

          Show
          busbey Sean Busbey added a comment - The change to a pom artifact was on purpose. My mistake missing a dependency that relied on the jar version. Andrew Wang , preference between an ADDENDUM patch and a new jira?
          Hide
          busbey Sean Busbey added a comment -

          Thinking about this more, the change to a pom type instead of an empty jar is better maven hygiene but means that downstream users who don't want to opt-in to the shaded dependencies will have to update their dependency declaration for the change.

          i.e.

          <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
          </dependency>
          

          has to change to

          <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <type>pom</type>
          </dependency>
          

          Is that what we want? On the positive side, it means downstream folks have to think about wether they want to keep using the client-with-third-party-dependencies or switch to the shaded client. On the negative side, we are forcing all downstream users to think about things instead of just those who are interested in the shaded client.

          Show
          busbey Sean Busbey added a comment - Thinking about this more, the change to a pom type instead of an empty jar is better maven hygiene but means that downstream users who don't want to opt-in to the shaded dependencies will have to update their dependency declaration for the change. i.e. <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> </dependency> has to change to <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <type>pom</type> </dependency> Is that what we want? On the positive side, it means downstream folks have to think about wether they want to keep using the client-with-third-party-dependencies or switch to the shaded client. On the negative side, we are forcing all downstream users to think about things instead of just those who are interested in the shaded client.
          Hide
          busbey Sean Busbey added a comment -

          follow-on JIRAs discussed above now linked as "is related to".

          Show
          busbey Sean Busbey added a comment - follow-on JIRAs discussed above now linked as "is related to".
          Hide
          jpallas Joe Pallas added a comment -

          I opened HADOOP-13922 because we didn't have any open bug capturing that the build really is broken. No criticism of Sean Busbey intended.

          Show
          jpallas Joe Pallas added a comment - I opened HADOOP-13922 because we didn't have any open bug capturing that the build really is broken. No criticism of Sean Busbey intended.
          Hide
          busbey Sean Busbey added a comment -

          no worries, I'll move my question over there.

          Show
          busbey Sean Busbey added a comment - no worries, I'll move my question over there.
          Hide
          anu Anu Engineer added a comment -

          Sean Busbey Thanks for all the hard work. I really appreciate it. I wanted to ask you a related question. I build hadoop all the time and with this patch the amount of time to build trunk has gone up from under 2 mins to 5+ mins on the same hardware. Do you see the same behaviour ? I just wanted to check this was not something on my hardware.

          if it is indeed due to shading, is there a way to switch the shading off or have this come into play only when we do a release build. The amount of time spent waiting for a build is a massive hit on productivity.

          So any tips on how I can get the old build speed back ?

          Thanks
          Anu

          Show
          anu Anu Engineer added a comment - Sean Busbey Thanks for all the hard work. I really appreciate it. I wanted to ask you a related question. I build hadoop all the time and with this patch the amount of time to build trunk has gone up from under 2 mins to 5+ mins on the same hardware. Do you see the same behaviour ? I just wanted to check this was not something on my hardware. if it is indeed due to shading, is there a way to switch the shading off or have this come into play only when we do a release build. The amount of time spent waiting for a build is a massive hit on productivity. So any tips on how I can get the old build speed back ? Thanks Anu
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          I assume it would take a new Maven profile to selectively skip shading. Seeing a 2x-3x increase in build times with the 'package' target. Twice, the build just stalled. I think it was stalled at the shading step from the build output.

          Show
          arpitagarwal Arpit Agarwal added a comment - I assume it would take a new Maven profile to selectively skip shading. Seeing a 2x-3x increase in build times with the 'package' target. Twice, the build just stalled. I think it was stalled at the shading step from the build output.
          Hide
          busbey Sean Busbey added a comment - - edited

          It would be unwise for us to add a profile to selectively have folks opt-in to producing the shaded artifacts. In Apache HBase that's been a source of build errors where folks end up with empty jars in nexus repos. Here in Hadoop we've been incrementally adding pre-commit coverage for profiles as folks realize they exist and aren't known to pre-commit.

          If you don't want to build the shaded artifacts, you can either run just the test goal, or ask maven to only build the module(s) you care about and any modules needed to build those module(s). e.g. mvn --projects :hadoop-yarn-project --also-make package or mvn --projects :hadoop-yarn-ui,:hadoop-yarn-server-timelineservice --also-make package. (You could also ask it to make modules that depend on your changed module with --also-make-dependents but that will almost certainly lead to building the shaded artifacts.)

          We could add a profile to selectively opt-out of creating the shaded artifacts, but the above maven commands are how one would do it "the maven way".

          Show
          busbey Sean Busbey added a comment - - edited It would be unwise for us to add a profile to selectively have folks opt-in to producing the shaded artifacts. In Apache HBase that's been a source of build errors where folks end up with empty jars in nexus repos. Here in Hadoop we've been incrementally adding pre-commit coverage for profiles as folks realize they exist and aren't known to pre-commit. If you don't want to build the shaded artifacts, you can either run just the test goal, or ask maven to only build the module(s) you care about and any modules needed to build those module(s). e.g. mvn --projects :hadoop-yarn-project --also-make package or mvn --projects :hadoop-yarn-ui,:hadoop-yarn-server-timelineservice --also-make package . (You could also ask it to make modules that depend on your changed module with --also-make-dependents but that will almost certainly lead to building the shaded artifacts.) We could add a profile to selectively opt-out of creating the shaded artifacts, but the above maven commands are how one would do it "the maven way".
          Hide
          kasha Karthik Kambatla added a comment -

          A profile to opt-out will be very useful.

          Also, is it possible to shade only on package and not on install?

          Show
          kasha Karthik Kambatla added a comment - A profile to opt-out will be very useful. Also, is it possible to shade only on package and not on install?
          Hide
          busbey Sean Busbey added a comment -

          Also, is it possible to shade only on package and not on install?

          Not as far as I know. The maven lifecycle specifies that the install goal requires the package goal to have run (ref).

          A profile to opt-out will be very useful.

          Can one of the three of y'all open a JIRA with the goals specified (i.e. "instructions added to XXXX for developers who want to skip shaded artifact generation") and assign it to me? Once I have a goal location for instructions I'll take on verifying some opt-out mechanism and documentation for same.

          Show
          busbey Sean Busbey added a comment - Also, is it possible to shade only on package and not on install? Not as far as I know. The maven lifecycle specifies that the install goal requires the package goal to have run ( ref ). A profile to opt-out will be very useful. Can one of the three of y'all open a JIRA with the goals specified (i.e. "instructions added to XXXX for developers who want to skip shaded artifact generation") and assign it to me? Once I have a goal location for instructions I'll take on verifying some opt-out mechanism and documentation for same.
          Hide
          cnauroth Chris Nauroth added a comment -

          I have +1'd the follow-up bug fix patch on HADOOP-13922, but I'm going to hold off committing since it's right before a holiday and others might be offline. I plan to commit it on Monday, 1/2.

          Show
          cnauroth Chris Nauroth added a comment - I have +1'd the follow-up bug fix patch on HADOOP-13922 , but I'm going to hold off committing since it's right before a holiday and others might be offline. I plan to commit it on Monday, 1/2.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          is this patch the cause of some dependency-reduced-pom.xml files appearing in my source? If so, they need adding to .gitignore

          Show
          stevel@apache.org Steve Loughran added a comment - is this patch the cause of some dependency-reduced-pom.xml files appearing in my source? If so, they need adding to .gitignore
          Hide
          andrew.wang Andrew Wang added a comment -

          This should be covered by the initial commit, I see an entry for dependency-reduced-pom.xml in the top-level .gitignore. I did a full build and "git status" was clean.

          If this still crops up, could you provide additional repro steps?

          Show
          andrew.wang Andrew Wang added a comment - This should be covered by the initial commit, I see an entry for dependency-reduced-pom.xml in the top-level .gitignore. I did a full build and "git status" was clean. If this still crops up, could you provide additional repro steps?
          Hide
          busbey Sean Busbey added a comment -

          thanks for checking Andrew Wang, I was pretty sure I had added things to .gitignore but hadn't had a chance to follow up yet.

          Steve Loughran mind filing a follow up with the repro steps (ala HADOOP-13922)? If you assign it to me I'd be happy to chase it down.

          Show
          busbey Sean Busbey added a comment - thanks for checking Andrew Wang , I was pretty sure I had added things to .gitignore but hadn't had a chance to follow up yet. Steve Loughran mind filing a follow up with the repro steps (ala HADOOP-13922 )? If you assign it to me I'd be happy to chase it down.
          Hide
          andrew.wang Andrew Wang added a comment -

          Sean Busbey would you mind if I moved the follow-on JIRAs for the shaded client to subtasks of HADOOP-11656? It would be nice for tracking purposes to have an open umbrella JIRA, and this one has been resolved.

          Show
          andrew.wang Andrew Wang added a comment - Sean Busbey would you mind if I moved the follow-on JIRAs for the shaded client to subtasks of HADOOP-11656 ? It would be nice for tracking purposes to have an open umbrella JIRA, and this one has been resolved.
          Hide
          busbey Sean Busbey added a comment -

          +1 that's a good idea.

          Show
          busbey Sean Busbey added a comment - +1 that's a good idea.
          Hide
          andrew.wang Andrew Wang added a comment -

          Done and done!

          Show
          andrew.wang Andrew Wang added a comment - Done and done!
          Hide
          andrew.wang Andrew Wang added a comment -

          Adding a release note so this pops up in alpha2. Sean, please feel free to update with any corrections.

          Show
          andrew.wang Andrew Wang added a comment - Adding a release note so this pops up in alpha2. Sean, please feel free to update with any corrections.

            People

            • Assignee:
              busbey Sean Busbey
              Reporter:
              busbey Sean Busbey
            • Votes:
              1 Vote for this issue
              Watchers:
              34 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development