Hadoop Common
  1. Hadoop Common
  2. HADOOP-8745

Incorrect version numbers in hadoop-core POM

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.0.3
    • Fix Version/s: 1.1.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The hadoop-core POM as published to Maven central has different dependency versions than Hadoop actually has on its runtime classpath. This can lead to client code working in unit tests but failing on the cluster and vice versa.

      The following version numbers are incorrect: jackson-mapper-asl, kfs, and jets3t. There's also a duplicate dependency to commons-net.

        Activity

        Hide
        Matthias Friedrich added a comment -

        Patch against branch-1.0. I wasn't able to run the Jenkins test locally, the instructions in HowToContribute seem to be for Hadoop 2. There isn't anything testable in the patch anyway.

        Show
        Matthias Friedrich added a comment - Patch against branch-1.0. I wasn't able to run the Jenkins test locally, the instructions in HowToContribute seem to be for Hadoop 2. There isn't anything testable in the patch anyway.
        Hide
        Eli Collins added a comment -

        Thanks Matthias.

        Is net.sf.kosmosfs actually being used anywhere in branch-1? The only imports are to org.kosmix, and per the following in ivy/hadoop-core.pom I think we can remove it.

            <!--Kosmos filesystem
            http://kosmosfs.sourceforge.net/
            This is not in the central repository
            -->
            <!--
                <dependency>
                  <groupId>org.kosmix</groupId>
                  <artifactId>kfs</artifactId>
                  <version>0.1</version>
                  <scope>optional</scope>
                </dependency>
            -->
        

        Otherwise the patch looks great!

        Show
        Eli Collins added a comment - Thanks Matthias. Is net.sf.kosmosfs actually being used anywhere in branch-1? The only imports are to org.kosmix, and per the following in ivy/hadoop-core.pom I think we can remove it. <!--Kosmos filesystem http: //kosmosfs.sourceforge.net/ This is not in the central repository --> <!-- <dependency> <groupId>org.kosmix</groupId> <artifactId>kfs</artifactId> <version>0.1</version> <scope>optional</scope> </dependency> --> Otherwise the patch looks great!
        Hide
        Matthias Friedrich added a comment -

        Thanks for your review! I don't know kfs, but I think there's some confusion because Hadoop's kfs-0.2.2 uses org.kosmix as the groupId while kfs-0.3 uses net.sf.kosmosfs as the groupId. It's the same package though, in Java they both use the org.kosmix package name space. It's commented in the ivy config because kfs-0.2.2 and older aren't available through Maven central. This is why the lib is checked into svn.

        The problem with my patch is that we can't downgrade to kfs-0.2.2 without causing problems for Maven users. There are three options: 1) leave it at 0.3, 2) deploy kfs 0.2.2 to Maven central, or 3) upgrade Hadoop to use kfs-0.3. Option 3 sounds like the best but I have no way of testing this (there's no test case and wiki documentation seems to have moved on to Hadoop 2). It compiles with some warnings, but whether it actually works is a different question.

        Show
        Matthias Friedrich added a comment - Thanks for your review! I don't know kfs, but I think there's some confusion because Hadoop's kfs-0.2.2 uses org.kosmix as the groupId while kfs-0.3 uses net.sf.kosmosfs as the groupId. It's the same package though, in Java they both use the org.kosmix package name space. It's commented in the ivy config because kfs-0.2.2 and older aren't available through Maven central. This is why the lib is checked into svn. The problem with my patch is that we can't downgrade to kfs-0.2.2 without causing problems for Maven users. There are three options: 1) leave it at 0.3, 2) deploy kfs 0.2.2 to Maven central, or 3) upgrade Hadoop to use kfs-0.3. Option 3 sounds like the best but I have no way of testing this (there's no test case and wiki documentation seems to have moved on to Hadoop 2). It compiles with some warnings, but whether it actually works is a different question.
        Hide
        Eli Collins added a comment -

        How about just removing the kosmosfs section from the pom entirely? If we use 0.2.2 and it's not available then that seems OK. kfs is an optional component anyway.

        Show
        Eli Collins added a comment - How about just removing the kosmosfs section from the pom entirely? If we use 0.2.2 and it's not available then that seems OK. kfs is an optional component anyway.
        Hide
        Matthias Friedrich added a comment -

        Yeah, I think that's a good idea, thanks! I've updated the patch.

        Show
        Matthias Friedrich added a comment - Yeah, I think that's a good idea, thanks! I've updated the patch.
        Hide
        Eli Collins added a comment -

        +1 lgtm, I've verified the build contains versions as specified.

        Show
        Eli Collins added a comment - +1 lgtm, I've verified the build contains versions as specified.
        Hide
        Eli Collins added a comment -

        I've committed this to branch-1. Thanks Matthias!

        Show
        Eli Collins added a comment - I've committed this to branch-1. Thanks Matthias!
        Hide
        Matt Foley added a comment -

        Merged to branch-1.1.

        Show
        Matt Foley added a comment - Merged to branch-1.1.
        Hide
        Matt Foley added a comment -

        Closed upon release of 1.1.1.

        Show
        Matt Foley added a comment - Closed upon release of 1.1.1.
        Hide
        Chris Nauroth added a comment -

        Could we please commit this fix to branch-1-win too? Thanks!

        Show
        Chris Nauroth added a comment - Could we please commit this fix to branch-1-win too? Thanks!
        Hide
        Suresh Srinivas added a comment -

        I have merged the patch to branch-1-win.

        Show
        Suresh Srinivas added a comment - I have merged the patch to branch-1-win.

          People

          • Assignee:
            Matthias Friedrich
            Reporter:
            Matthias Friedrich
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development