Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13922

Some modules have dependencies on hadoop-client jar removed by HADOOP-11804

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: build
    • Labels:
      None

      Description

      As discussed in HADOOP-11804 comment 15758048 and following comments, there are still dependencies on the now-removed hadoop-client jar. The current code builds only because an obsolete snapshot of the jar is found on the repository server. Changing the project version to something new exposes the problem.

      While the build currently dies at hadoop-tools/hadoop-sls, I'm seeing issues with some Hadoop Client modules, too.

      I'm filing a new bug because I can't reopen HADOOP-11804.

        Issue Links

          Activity

          Hide
          busbey Sean Busbey added a comment -

          pulled from my question on HADOOP-11804:

          Thinking about this more, the change to a pom type instead of an empty jar is better maven hygiene but means that downstream users who don't want to opt-in to the shaded dependencies will have to update their dependency declaration for the change.
          i.e.

          <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
          </dependency>
          

          has to change to

          <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <type>pom</type>
          </dependency>
          

          Is that what we want? On the positive side, it means downstream folks have to think about wether they want to keep using the client-with-third-party-dependencies or switch to the shaded client. On the negative side, we are forcing all downstream users to think about things instead of just those who are interested in the shaded client.

          Happy to either clean up references to the non-pom dependency or change hadoop-client back to an empty jar. Just need to know which a committer would prefer.

          (If I don't hear back one way or the other tonight, I'll decide which I think is better and put up a patch + reasoning tomorrow.)

          Show
          busbey Sean Busbey added a comment - pulled from my question on HADOOP-11804 : Thinking about this more, the change to a pom type instead of an empty jar is better maven hygiene but means that downstream users who don't want to opt-in to the shaded dependencies will have to update their dependency declaration for the change. i.e. <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> </dependency> has to change to <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <type>pom</type> </dependency> Is that what we want? On the positive side, it means downstream folks have to think about wether they want to keep using the client-with-third-party-dependencies or switch to the shaded client. On the negative side, we are forcing all downstream users to think about things instead of just those who are interested in the shaded client. Happy to either clean up references to the non-pom dependency or change hadoop-client back to an empty jar. Just need to know which a committer would prefer. (If I don't hear back one way or the other tonight, I'll decide which I think is better and put up a patch + reasoning tomorrow.)
          Hide
          busbey Sean Busbey added a comment -

          -01

          • changes hadoop-client packaging back to jar.

          Here's my reasoning: keeping things as jar packaging means there's no impact for downstream clients who upgrade from hadoop 2.y to 3.y. Additionally, as the author of HADOOP-11804 and a person who's done a fair bit of maven build work, I failed to properly use the pom-type when specifying hadoop-client as a dependency everywhere in the project. :/ That reads pretty clearly to me as "downstream users are unlikely to grok a benefit for changing to pom packaging".

          Show
          busbey Sean Busbey added a comment - -01 changes hadoop-client packaging back to jar. Here's my reasoning: keeping things as jar packaging means there's no impact for downstream clients who upgrade from hadoop 2.y to 3.y. Additionally, as the author of HADOOP-11804 and a person who's done a fair bit of maven build work, I failed to properly use the pom-type when specifying hadoop-client as a dependency everywhere in the project. :/ That reads pretty clearly to me as "downstream users are unlikely to grok a benefit for changing to pom packaging".
          Hide
          busbey Sean Busbey added a comment -

          As a side comment, if we don't go with the approach of changing hadoop-client back to jar packaging then we should flag HADOOP-11804 as incompatible.

          Show
          busbey Sean Busbey added a comment - As a side comment, if we don't go with the approach of changing hadoop-client back to jar packaging then we should flag HADOOP-11804 as incompatible.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 14m 45s trunk passed
          +1 compile 0m 48s trunk passed
          +1 mvnsite 0m 53s trunk passed
          +1 mvneclipse 0m 48s trunk passed
          +1 javadoc 0m 43s trunk passed
          +1 mvninstall 0m 32s the patch passed
          +1 compile 0m 28s the patch passed
          +1 javac 0m 28s the patch passed
          +1 mvnsite 0m 35s the patch passed
          +1 mvneclipse 0m 33s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          +1 javadoc 0m 27s the patch passed
          +1 unit 0m 32s hadoop-client in the patch passed.
          +1 asflicense 0m 56s The patch does not generate ASF License warnings.
          24m 12s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13922
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844997/HADOOP-13922.1.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux bc53e608e003 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f216276
          Default Java 1.8.0_111
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11331/testReport/
          modules C: hadoop-client-modules/hadoop-client U: hadoop-client-modules/hadoop-client
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11331/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 14m 45s trunk passed +1 compile 0m 48s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 48s trunk passed +1 javadoc 0m 43s trunk passed +1 mvninstall 0m 32s the patch passed +1 compile 0m 28s the patch passed +1 javac 0m 28s the patch passed +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. +1 javadoc 0m 27s the patch passed +1 unit 0m 32s hadoop-client in the patch passed. +1 asflicense 0m 56s The patch does not generate ASF License warnings. 24m 12s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13922 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844997/HADOOP-13922.1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux bc53e608e003 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f216276 Default Java 1.8.0_111 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11331/testReport/ modules C: hadoop-client-modules/hadoop-client U: hadoop-client-modules/hadoop-client Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11331/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jpallas Joe Pallas added a comment -

          I confirmed that a build with a novel version number works with this patch.

          With regard to the packaging question, my Maven experience is pretty vanilla, and I had to read up in the Sonatype book to get a better understanding of when and how pom packaging should be used. To be honest, that left me with very little understanding/motivation for using pom packaging outside of inter-module dependencies within a single project.

          Show
          jpallas Joe Pallas added a comment - I confirmed that a build with a novel version number works with this patch. With regard to the packaging question, my Maven experience is pretty vanilla, and I had to read up in the Sonatype book to get a better understanding of when and how pom packaging should be used. To be honest, that left me with very little understanding/motivation for using pom packaging outside of inter-module dependencies within a single project.
          Hide
          busbey Sean Busbey added a comment -

          Yeah. The dev commentary on extra-enforcer-rules #13 also left me with little confidence in the overall maven ecosystem's ability to cope with pom dependencies, which has been an issue even for inter-module dependencies within HBase (HBASE-17286).

          Thanks for checking the fix Joe Pallas!

          I think this is ready to go.

          Show
          busbey Sean Busbey added a comment - Yeah. The dev commentary on extra-enforcer-rules #13 also left me with little confidence in the overall maven ecosystem's ability to cope with pom dependencies, which has been an issue even for inter-module dependencies within HBase ( HBASE-17286 ). Thanks for checking the fix Joe Pallas ! I think this is ready to go.
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 for the patch. I had a clean new dev box that was hitting this build problem, and I confirmed that this patch fixes it. I think it's the right thing to do in the interest of compatibility with existing downstream consumers.

          Since it's right before a holiday and interested people might be offline partying hard, I will hold off the commit. I plan to commit this on Monday, 1/2, unless I hear otherwise.

          Show
          cnauroth Chris Nauroth added a comment - +1 for the patch. I had a clean new dev box that was hitting this build problem, and I confirmed that this patch fixes it. I think it's the right thing to do in the interest of compatibility with existing downstream consumers. Since it's right before a holiday and interested people might be offline partying hard, I will hold off the commit. I plan to commit this on Monday, 1/2, unless I hear otherwise.
          Hide
          mblo Marcel B added a comment -

          The patch works for me (mvn package), however mvn eclipse:eclipse still fails with same error. I had a clean new env (osx 10.12.2, jdk8). I'll provide more details on request.

          Show
          mblo Marcel B added a comment - The patch works for me (mvn package), however mvn eclipse:eclipse still fails with same error. I had a clean new env (osx 10.12.2, jdk8). I'll provide more details on request.
          Hide
          busbey Sean Busbey added a comment -

          Marcel B please provide more context; the precommit test above shows the eclipse command working.

          Show
          busbey Sean Busbey added a comment - Marcel B please provide more context; the precommit test above shows the eclipse command working.
          Hide
          cnauroth Chris Nauroth added a comment -

          I have committed this to trunk. Sean Busbey, thank you for contributing the fix.

          Marcel B, I chose to commit this, because it fixes the immediate problem that is impacting some of us. If you continue to reproduce issues running mvn eclipse:eclipse, then please feel free to file a new JIRA issue with the details. Thank you.

          Show
          cnauroth Chris Nauroth added a comment - I have committed this to trunk. Sean Busbey , thank you for contributing the fix. Marcel B , I chose to commit this, because it fixes the immediate problem that is impacting some of us. If you continue to reproduce issues running mvn eclipse:eclipse , then please feel free to file a new JIRA issue with the details. Thank you.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11066 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11066/)
          HADOOP-13922. Some modules have dependencies on hadoop-client jar (cnauroth: rev ebdd2e03b7b8573cc3531958dbfda72cdbc277fd)

          • (edit) hadoop-client-modules/hadoop-client/pom.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11066 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11066/ ) HADOOP-13922 . Some modules have dependencies on hadoop-client jar (cnauroth: rev ebdd2e03b7b8573cc3531958dbfda72cdbc277fd) (edit) hadoop-client-modules/hadoop-client/pom.xml
          Hide
          mblo Marcel B added a comment -

          Sorry for the noise - updating b31e195..e49e0a6 solved the issue. I have no clue what was going on yesterday. Thank you!

          Show
          mblo Marcel B added a comment - Sorry for the noise - updating b31e195..e49e0a6 solved the issue. I have no clue what was going on yesterday. Thank you!
          Hide
          cnauroth Chris Nauroth added a comment -

          Marcel B, no worries, and thank you for coming back to confirm that the problem no longer repros.

          Show
          cnauroth Chris Nauroth added a comment - Marcel B , no worries, and thank you for coming back to confirm that the problem no longer repros.

            People

            • Assignee:
              busbey Sean Busbey
              Reporter:
              jpallas Joe Pallas
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development