Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13989

Remove erroneous source jar option from hadoop-client shade configuration

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: 3.0.0-alpha4
    • Component/s: build
    • Labels:
      None

      Description

      The pom files for hadoop-client-minicluster and hadoop-client-runtime have a typo in the configuration of the shade module. They say <createSourceJar> instead of <createSourcesJar>. (This was noticed by IntelliJ, but not by maven.)

      Shade plugin doc is at http://maven.apache.org/plugins/maven-shade-plugin/shade-mojo.html#createSourcesJar.

      1. HADOOP-13989.001.patch
        2 kB
        Joe Pallas
      2. HADOOP-13989.002.patch
        1 kB
        Joe Pallas
      3. HADOOP-13989.003.patch
        1 kB
        Joe Pallas

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11176 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11176/)
        HADOOP-13989. Remove erroneous source jar option from hadoop-client (wang: rev cd59b9ccab51376310484a6e3d9179bb52fccae1)

        • (edit) hadoop-client-modules/hadoop-client-minicluster/pom.xml
        • (edit) hadoop-client-modules/hadoop-client-runtime/pom.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11176 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11176/ ) HADOOP-13989 . Remove erroneous source jar option from hadoop-client (wang: rev cd59b9ccab51376310484a6e3d9179bb52fccae1) (edit) hadoop-client-modules/hadoop-client-minicluster/pom.xml (edit) hadoop-client-modules/hadoop-client-runtime/pom.xml
        Hide
        andrew.wang Andrew Wang added a comment -

        Committed to trunk, thanks Joe for the patch, Sean for reviewing!

        Show
        andrew.wang Andrew Wang added a comment - Committed to trunk, thanks Joe for the patch, Sean for reviewing!
        Hide
        busbey Sean Busbey added a comment -

        +1 on v3.

        Show
        busbey Sean Busbey added a comment - +1 on v3.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 11s Maven dependency ordering for branch
        +1 mvninstall 12m 39s trunk passed
        +1 compile 0m 16s trunk passed
        +1 mvnsite 0m 26s trunk passed
        +1 mvneclipse 0m 28s trunk passed
        +1 javadoc 0m 19s trunk passed
        0 mvndep 0m 7s Maven dependency ordering for patch
        +1 mvninstall 4m 46s the patch passed
        +1 compile 0m 13s the patch passed
        +1 javac 0m 13s the patch passed
        +1 mvnsite 0m 21s the patch passed
        +1 mvneclipse 0m 21s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        +1 javadoc 0m 14s the patch passed
        +1 unit 0m 8s hadoop-client-runtime in the patch passed.
        +1 unit 0m 10s hadoop-client-minicluster in the patch passed.
        +1 asflicense 0m 14s The patch does not generate ASF License warnings.
        21m 49s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HADOOP-13989
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848387/HADOOP-13989.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 71274503bfc6 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f5839fd
        Default Java 1.8.0_111
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11472/testReport/
        modules C: hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster U: hadoop-client-modules
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11472/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 12m 39s trunk passed +1 compile 0m 16s trunk passed +1 mvnsite 0m 26s trunk passed +1 mvneclipse 0m 28s trunk passed +1 javadoc 0m 19s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 4m 46s the patch passed +1 compile 0m 13s the patch passed +1 javac 0m 13s the patch passed +1 mvnsite 0m 21s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 javadoc 0m 14s the patch passed +1 unit 0m 8s hadoop-client-runtime in the patch passed. +1 unit 0m 10s hadoop-client-minicluster in the patch passed. +1 asflicense 0m 14s The patch does not generate ASF License warnings. 21m 49s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13989 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848387/HADOOP-13989.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 71274503bfc6 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f5839fd Default Java 1.8.0_111 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11472/testReport/ modules C: hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster U: hadoop-client-modules Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11472/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jpallas Joe Pallas added a comment -

        Needed rebase after HADOOP-13999

        Show
        jpallas Joe Pallas added a comment - Needed rebase after HADOOP-13999
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 13s HADOOP-13989 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Issue HADOOP-13989
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848375/HADOOP-13989.002.patch
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11469/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 13s HADOOP-13989 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HADOOP-13989 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848375/HADOOP-13989.002.patch Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11469/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jpallas Joe Pallas added a comment - - edited

        Repeal now, replace later (new patch)

        Show
        jpallas Joe Pallas added a comment - - edited Repeal now, replace later (new patch)
        Hide
        busbey Sean Busbey added a comment -

        I'm not sure about the JIRA protocol here: would we rename this JIRA to something like "Remove erroneous source jar option from hadoop-client shade configuration" and open a new one for something like "Create shaded sources jars in hadoop-client modules"?

        yeah, sounds good.

        Show
        busbey Sean Busbey added a comment - I'm not sure about the JIRA protocol here: would we rename this JIRA to something like "Remove erroneous source jar option from hadoop-client shade configuration" and open a new one for something like "Create shaded sources jars in hadoop-client modules"? yeah, sounds good.
        Hide
        andrew.wang Andrew Wang added a comment -

        I think it's fine to reuse this one, we have enough watchers that we'll get a review in promptly.

        Show
        andrew.wang Andrew Wang added a comment - I think it's fine to reuse this one, we have enough watchers that we'll get a review in promptly.
        Hide
        jpallas Joe Pallas added a comment -

        Agreed, the current createSourceJar option does nothing (except possibly confuse people who are looking for source jars), so removing it seems reasonable.

        I'm not sure about the JIRA protocol here: would we rename this JIRA to something like "Remove erroneous source jar option from hadoop-client shade configuration" and open a new one for something like "Create shaded sources jars in hadoop-client modules"?

        Show
        jpallas Joe Pallas added a comment - Agreed, the current createSourceJar option does nothing (except possibly confuse people who are looking for source jars), so removing it seems reasonable. I'm not sure about the JIRA protocol here: would we rename this JIRA to something like "Remove erroneous source jar option from hadoop-client shade configuration" and open a new one for something like "Create shaded sources jars in hadoop-client modules"?
        Hide
        busbey Sean Busbey added a comment -

        We could start by just removing the source jar option entirely, since that's equivalent output. without the source jars IDEs can't auto-populate the source when folks use the shaded dependencies, so we should try to fix it at some point.

        Show
        busbey Sean Busbey added a comment - We could start by just removing the source jar option entirely, since that's equivalent output. without the source jars IDEs can't auto-populate the source when folks use the shaded dependencies, so we should try to fix it at some point.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Sorry for breaking the post commit job and thanks Andrew for reverting this.

        Show
        ajisakaa Akira Ajisaka added a comment - Sorry for breaking the post commit job and thanks Andrew for reverting this.
        Hide
        jpallas Joe Pallas added a comment -

        So far my experiments with <forceCreation>true</forceCreation> have not been successful, although the shade plugin is cited by the jar plugin documentation for forceCreation as a use case. I'm guessing that something about the handling of sources jars is different or broken.

        Show
        jpallas Joe Pallas added a comment - So far my experiments with <forceCreation>true</forceCreation> have not been successful, although the shade plugin is cited by the jar plugin documentation for forceCreation as a use case. I'm guessing that something about the handling of sources jars is different or broken.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11131 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11131/)
        Revert "HADOOP-13989. Fix typo in hadoop-client shade configuration. (wang: rev 47dfda7c92bf6c1b460540758412a7a31b4bee54)

        • (edit) hadoop-client-modules/hadoop-client-minicluster/pom.xml
        • (edit) hadoop-client-modules/hadoop-client-runtime/pom.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11131 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11131/ ) Revert " HADOOP-13989 . Fix typo in hadoop-client shade configuration. (wang: rev 47dfda7c92bf6c1b460540758412a7a31b4bee54) (edit) hadoop-client-modules/hadoop-client-minicluster/pom.xml (edit) hadoop-client-modules/hadoop-client-runtime/pom.xml
        Hide
        andrew.wang Andrew Wang added a comment -

        FYI that I've reverted this from trunk for now.

        Show
        andrew.wang Andrew Wang added a comment - FYI that I've reverted this from trunk for now.
        Hide
        busbey Sean Busbey added a comment -

        Can we use the work-around suggested in MSHADE-126?

        Adding <forceCreation>true</forceCreation> to the maven-jar-plugin works around this.

        Show
        busbey Sean Busbey added a comment - Can we use the work-around suggested in MSHADE-126 ? Adding <forceCreation>true</forceCreation> to the maven-jar-plugin works around this.
        Hide
        jpallas Joe Pallas added a comment -

        Sean Busbey, it looks like there is such a known bug in the shade plugin (MSHADE-126, resolved as Won't Fix). The workaround is to force the jar file to be recreated every time. Doing clean install followed by deploy did reproduce the problem in my environment, so at least I can verify an attempted solution.

        Show
        jpallas Joe Pallas added a comment - Sean Busbey , it looks like there is such a known bug in the shade plugin ( MSHADE-126 , resolved as Won't Fix). The workaround is to force the jar file to be recreated every time. Doing clean install followed by deploy did reproduce the problem in my environment, so at least I can verify an attempted solution.
        Hide
        bibinchundatt Bibin A Chundatt added a comment - - edited

        Joe Pallas
        Tried reverting HADOOP-13989 locally looks like its working fine.
        Locally i tried package to reproduce

        mvn package -Pdist -Dtar -Pnative -Dmaven.javadoc.skip=true -Dmaven.repo.local=/opt/apacheprojects/hadoop/repository -Pyarn-ui -DskipTests  -rf :hadoop-client-runtime --debug
        
        hadoop.shaded.com.fasterxml.jackson.databind.Module -> [Help 1]
        org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:2.4.3:shade (default) on project hadoop-client-runtime: Error creating shaded jar: duplicate entry: META-INF/services/org.apache.hadoop.shaded.com.fasterxml.jackson.databind.Module
        	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
        	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
        	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
        	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
        	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
        	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
        	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
        	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
        	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
        	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
        	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
        	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
        	at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:498)
        	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
        	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
        	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
        	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
        Caused by: org.apache.maven.plugin.MojoExecutionException: Error creating shaded jar: duplicate entry: META-INF/services/org.apache.hadoop.shaded.com.fasterxml.jackson.databind.Module
        	at org.apache.maven.plugins.shade.mojo.ShadeMojo.execute(ShadeMojo.java:540)
        	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
        	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
        	... 20 more
        Caused by: java.util.zip.ZipException: duplicate entry: META-INF/services/org.apache.hadoop.shaded.com.fasterxml.jackson.databind.Module
        	at java.util.zip.ZipOutputStream.putNextEntry(ZipOutputStream.java:232)
        	at java.util.jar.JarOutputStream.putNextEntry(JarOutputStream.java:109)
        	at org.apache.hadoop.maven.plugin.shade.resource.ServicesResourceTransformer.modifyOutputStream(ServicesResourceTransformer.java:110)
        	at org.apache.maven.plugins.shade.DefaultShader.shade(DefaultShader.java:131)
        	at org.apache.maven.plugins.shade.mojo.ShadeMojo.execute(ShadeMojo.java:461)
        	... 22 more
        
        
        Show
        bibinchundatt Bibin A Chundatt added a comment - - edited Joe Pallas Tried reverting HADOOP-13989 locally looks like its working fine. Locally i tried package to reproduce mvn package -Pdist -Dtar -Pnative -Dmaven.javadoc.skip= true -Dmaven.repo.local=/opt/apacheprojects/hadoop/repository -Pyarn-ui -DskipTests -rf :hadoop-client-runtime --debug hadoop.shaded.com.fasterxml.jackson.databind.Module -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:2.4.3:shade ( default ) on project hadoop-client-runtime: Error creating shaded jar: duplicate entry: META-INF/services/org.apache.hadoop.shaded.com.fasterxml.jackson.databind.Module at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) at org.apache.maven.cli.MavenCli.main(MavenCli.java:199) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) Caused by: org.apache.maven.plugin.MojoExecutionException: Error creating shaded jar: duplicate entry: META-INF/services/org.apache.hadoop.shaded.com.fasterxml.jackson.databind.Module at org.apache.maven.plugins.shade.mojo.ShadeMojo.execute(ShadeMojo.java:540) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207) ... 20 more Caused by: java.util.zip.ZipException: duplicate entry: META-INF/services/org.apache.hadoop.shaded.com.fasterxml.jackson.databind.Module at java.util.zip.ZipOutputStream.putNextEntry(ZipOutputStream.java:232) at java.util.jar.JarOutputStream.putNextEntry(JarOutputStream.java:109) at org.apache.hadoop.maven.plugin.shade.resource.ServicesResourceTransformer.modifyOutputStream(ServicesResourceTransformer.java:110) at org.apache.maven.plugins.shade.DefaultShader.shade(DefaultShader.java:131) at org.apache.maven.plugins.shade.mojo.ShadeMojo.execute(ShadeMojo.java:461) ... 22 more
        Hide
        busbey Sean Busbey added a comment -

        Here's what Jenkins is doing:

        ${MAVEN_HOME}/bin/mvn clean install \
        	-DskipTests -Pnative --threads .5C \
            -Dmaven.repo.local="${MAVEN_LOCAL_REPO}"
        ${MAVEN_HOME}/bin/mvn deploy -DdeployAtEnd=true \
        	-DretryFailedDeploymentCount=10 -DskipTests \
            -Dmaven.repo.local="${MAVEN_LOCAL_REPO}"
        

        The failure, as you mentioned, is in the second invocation that does deploy. I wonder if maybe it's a bug in the shade plugin that is trying to pack more stuff into the existing jar?

        Show
        busbey Sean Busbey added a comment - Here's what Jenkins is doing: ${MAVEN_HOME}/bin/mvn clean install \ -DskipTests -Pnative --threads .5C \ -Dmaven.repo.local= "${MAVEN_LOCAL_REPO}" ${MAVEN_HOME}/bin/mvn deploy -DdeployAtEnd= true \ -DretryFailedDeploymentCount=10 -DskipTests \ -Dmaven.repo.local= "${MAVEN_LOCAL_REPO}" The failure, as you mentioned, is in the second invocation that does deploy. I wonder if maybe it's a bug in the shade plugin that is trying to pack more stuff into the existing jar?
        Hide
        busbey Sean Busbey added a comment -

        looks like the post-commit job has run a few more times and consistently fails as here.

        Show
        busbey Sean Busbey added a comment - looks like the post-commit job has run a few more times and consistently fails as here.
        Hide
        busbey Sean Busbey added a comment -

        I hope we don't have any SNAPSHOT dependencies, so they shouldn't have changed. What command are you using to do the local repro? Has anyone re-run the post-commit job yet?

        Show
        busbey Sean Busbey added a comment - I hope we don't have any SNAPSHOT dependencies, so they shouldn't have changed. What command are you using to do the local repro? Has anyone re-run the post-commit job yet?
        Hide
        jpallas Joe Pallas added a comment -

        That's a fair point. Since I haven't been able to reproduce the problem, I'm not able to confirm that reversion fixes it. All I have to go on is that the last successful build was just before the change and the first failing build only included this and YARN-6057 (changes to yarn-default.xml). So, any changes in dependencies would have to be outside of hadoop, I think. Are there unstable dependencies?

        Show
        jpallas Joe Pallas added a comment - That's a fair point. Since I haven't been able to reproduce the problem, I'm not able to confirm that reversion fixes it. All I have to go on is that the last successful build was just before the change and the first failing build only included this and YARN-6057 (changes to yarn-default.xml ). So, any changes in dependencies would have to be outside of hadoop, I think. Are there unstable dependencies?
        Hide
        busbey Sean Busbey added a comment -

        I'd confirm that a revert fixes things locally before going that route.

        [ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:2.4.3:shade (default) on project hadoop-client-runtime: Error creating shaded jar: duplicate entry: META-INF/services/org.apache.hadoop.shaded.com.fasterxml.jackson.databind.Module -> [Help 1]
        

        this failure happened while building the binary jar, not while creating any source jar. It could just be coincidence that this patch showed up at the same time as something else that changed dependencies in a breaking way.

        Show
        busbey Sean Busbey added a comment - I'd confirm that a revert fixes things locally before going that route. [ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:2.4.3:shade ( default ) on project hadoop-client-runtime: Error creating shaded jar: duplicate entry: META-INF/services/org.apache.hadoop.shaded.com.fasterxml.jackson.databind.Module -> [Help 1] this failure happened while building the binary jar, not while creating any source jar. It could just be coincidence that this patch showed up at the same time as something else that changed dependencies in a breaking way.
        Hide
        jpallas Joe Pallas added a comment -

        The build failure that was reported appears during mvn deploy, but not during mvn install. This is puzzling, since the change is to the shade configuration, and shade is run during the packaging phase of the lifecycle. I haven't been able to reproduce the problem in my build environment, either. This seemingly trivial fix may actually require someone who is much more of a Maven wizard than I am. (Sean Busbey?)

        For now it looks like the only sensible thing is to revert the commit, Akira Ajisaka. Do you agree?

        Show
        jpallas Joe Pallas added a comment - The build failure that was reported appears during mvn deploy , but not during mvn install . This is puzzling, since the change is to the shade configuration, and shade is run during the packaging phase of the lifecycle. I haven't been able to reproduce the problem in my build environment, either. This seemingly trivial fix may actually require someone who is much more of a Maven wizard than I am. ( Sean Busbey ?) For now it looks like the only sensible thing is to revert the commit, Akira Ajisaka . Do you agree?
        Hide
        jpallas Joe Pallas added a comment -

        I don't get it. Pre-commit build was fine, but the build failure certainly seems to be connected to this change.

        Show
        jpallas Joe Pallas added a comment - I don't get it. Pre-commit build was fine, but the build failure certainly seems to be connected to this change.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11126 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11126/)
        HADOOP-13989. Fix typo in hadoop-client shade configuration. Contributed (aajisaka: rev 7e81063c13a636c551d3281966a24ccfcfd46793)

        • (edit) hadoop-client-modules/hadoop-client-minicluster/pom.xml
        • (edit) hadoop-client-modules/hadoop-client-runtime/pom.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11126 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11126/ ) HADOOP-13989 . Fix typo in hadoop-client shade configuration. Contributed (aajisaka: rev 7e81063c13a636c551d3281966a24ccfcfd46793) (edit) hadoop-client-modules/hadoop-client-minicluster/pom.xml (edit) hadoop-client-modules/hadoop-client-runtime/pom.xml
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Committed this to trunk. Thanks Joe Pallas for the contribution!
        FYI: The Fix Version/s is set by a committer when the patch is committed, so I'll appreciate if you don't set the fix version in the next time.

        Show
        ajisakaa Akira Ajisaka added a comment - Committed this to trunk. Thanks Joe Pallas for the contribution! FYI: The Fix Version/s is set by a committer when the patch is committed, so I'll appreciate if you don't set the fix version in the next time.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        +1, checking this in.

        Show
        ajisakaa Akira Ajisaka added a comment - +1, checking this in.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 26s Maven dependency ordering for branch
        +1 mvninstall 14m 26s trunk passed
        +1 compile 0m 17s trunk passed
        +1 mvnsite 0m 29s trunk passed
        +1 mvneclipse 0m 30s trunk passed
        +1 javadoc 0m 20s trunk passed
        0 mvndep 0m 7s Maven dependency ordering for patch
        +1 mvninstall 5m 51s the patch passed
        +1 compile 0m 15s the patch passed
        +1 javac 0m 15s the patch passed
        +1 mvnsite 0m 23s the patch passed
        +1 mvneclipse 0m 24s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 3s The patch has no ill-formed XML file.
        +1 javadoc 0m 15s the patch passed
        +1 unit 0m 9s hadoop-client-runtime in the patch passed.
        +1 unit 0m 11s hadoop-client-minicluster in the patch passed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        25m 19s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HADOOP-13989
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847454/HADOOP-13989.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 21a836a7e904 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d3170f9
        Default Java 1.8.0_111
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11438/testReport/
        modules C: hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster U: hadoop-client-modules
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11438/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 26s Maven dependency ordering for branch +1 mvninstall 14m 26s trunk passed +1 compile 0m 17s trunk passed +1 mvnsite 0m 29s trunk passed +1 mvneclipse 0m 30s trunk passed +1 javadoc 0m 20s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 5m 51s the patch passed +1 compile 0m 15s the patch passed +1 javac 0m 15s the patch passed +1 mvnsite 0m 23s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 3s The patch has no ill-formed XML file. +1 javadoc 0m 15s the patch passed +1 unit 0m 9s hadoop-client-runtime in the patch passed. +1 unit 0m 11s hadoop-client-minicluster in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 25m 19s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13989 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847454/HADOOP-13989.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 21a836a7e904 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d3170f9 Default Java 1.8.0_111 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11438/testReport/ modules C: hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster U: hadoop-client-modules Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11438/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

          People

          • Assignee:
            jpallas Joe Pallas
            Reporter:
            jpallas Joe Pallas
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development