Avro
  1. Avro
  2. AVRO-1410

Explicit version specification in pom prevents dependency management

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.7.5
    • Fix Version/s: 1.7.6
    • Component/s: java
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      While building Avro against different versions of Hadoop I found that avro-mapred explicitly specifies the versions of some of the Hadoop dependencies rather than deferring to its own dependency management. This makes it impossible to build against other versions of Hadoop without pom-hackery. It's also a potential maintenance issue for Avro in that changes to the dependency management section of the Java top level pom won't be picked up by the mapred module.

      1. AVRO-1410.patch
        2 kB
        E. Sammer
      2. AVRO-1410-2.patch
        4 kB
        E. Sammer

        Activity

        Hide
        E. Sammer added a comment -

        Attaching a patch. Passes all tests. I have a CLA on file with the ASF.

        Show
        E. Sammer added a comment - Attaching a patch. Passes all tests. I have a CLA on file with the ASF.
        Hide
        E. Sammer added a comment -

        Updating the patch to include all mapred dependencies rather than just the Hadoop 2.x deps.

        Show
        E. Sammer added a comment - Updating the patch to include all mapred dependencies rather than just the Hadoop 2.x deps.
        Hide
        Doug Cutting added a comment -

        We build different versions of the Avro mapred jar for different versions of Hadoop (where abstract classes changed to interfaces). See AVRO-1170. Unit tests do not check this. Are you certain your patch does not break this?

        Show
        Doug Cutting added a comment - We build different versions of the Avro mapred jar for different versions of Hadoop (where abstract classes changed to interfaces). See AVRO-1170 . Unit tests do not check this. Are you certain your patch does not break this?
        Hide
        Tom White added a comment -

        You can build against different versions of Hadoop by specifying -Dhadoop1.version=... for Hadoop 1, or -Dhadoop.version=2 -Dhadoop2.version=... for Hadoop 2.

        Also, Hadoop is a provided (or test) dependency for avro-mapred so you can use a different version of Hadoop at runtime, as long as it is the same major version (1 or 2).

        Show
        Tom White added a comment - You can build against different versions of Hadoop by specifying -Dhadoop1.version=... for Hadoop 1, or -Dhadoop.version=2 -Dhadoop2.version=... for Hadoop 2. Also, Hadoop is a provided (or test) dependency for avro-mapred so you can use a different version of Hadoop at runtime, as long as it is the same major version (1 or 2).
        Hide
        Doug Cutting added a comment -

        > You can build against different versions of Hadoop by specifying ...

        But will that still work after the proposed patch is applied? It's not clear to me that it will.

        Show
        Doug Cutting added a comment - > You can build against different versions of Hadoop by specifying ... But will that still work after the proposed patch is applied? It's not clear to me that it will.
        Hide
        Tom White added a comment -

        The patch simply moves the version numbers into the top-level Java POM's dependency management section (where other version numbers are already defined), rather than having them defined in the avro-mapred POM. The dependencies and version numbers for each profile (Hadoop 1 and 2) are unchanged. So I'm +1 on the change.

        I verified that running the tests under each profile was successful with the change:

        mvn clean install
        mvn -Dhadoop.version=2 clean install
        
        Show
        Tom White added a comment - The patch simply moves the version numbers into the top-level Java POM's dependency management section (where other version numbers are already defined), rather than having them defined in the avro-mapred POM. The dependencies and version numbers for each profile (Hadoop 1 and 2) are unchanged. So I'm +1 on the change. I verified that running the tests under each profile was successful with the change: mvn clean install mvn -Dhadoop.version=2 clean install
        Hide
        Doug Cutting added a comment -

        Thanks, Tom, for testing this. I was somehow mis-reading the patch. +1

        Show
        Doug Cutting added a comment - Thanks, Tom, for testing this. I was somehow mis-reading the patch. +1
        Hide
        ASF subversion and git services added a comment -

        Commit 1551255 from tomwhite@apache.org in branch 'avro/trunk'
        [ https://svn.apache.org/r1551255 ]

        AVRO-1410. Explicit version specification in pom prevents dependency management. Contributed by Eric Sammer.

        Show
        ASF subversion and git services added a comment - Commit 1551255 from tomwhite@apache.org in branch 'avro/trunk' [ https://svn.apache.org/r1551255 ] AVRO-1410 . Explicit version specification in pom prevents dependency management. Contributed by Eric Sammer.
        Hide
        ASF subversion and git services added a comment -

        Commit 1551257 from tomwhite@apache.org in branch 'avro/trunk'
        [ https://svn.apache.org/r1551257 ]

        Update CHANGES.txt for AVRO-1410.

        Show
        ASF subversion and git services added a comment - Commit 1551257 from tomwhite@apache.org in branch 'avro/trunk' [ https://svn.apache.org/r1551257 ] Update CHANGES.txt for AVRO-1410 .
        Hide
        Tom White added a comment -

        I committed this. Thanks Eric!

        Show
        Tom White added a comment - I committed this. Thanks Eric!
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in AvroJava #409 (See https://builds.apache.org/job/AvroJava/409/)
        Update CHANGES.txt for AVRO-1410. (tomwhite: rev 1551257)

        • /avro/trunk/CHANGES.txt
          AVRO-1410. Explicit version specification in pom prevents dependency management. Contributed by Eric Sammer. (tomwhite: rev 1551255)
        • /avro/trunk/lang/java/mapred/pom.xml
        • /avro/trunk/lang/java/pom.xml
        Show
        Hudson added a comment - SUCCESS: Integrated in AvroJava #409 (See https://builds.apache.org/job/AvroJava/409/ ) Update CHANGES.txt for AVRO-1410 . (tomwhite: rev 1551257) /avro/trunk/CHANGES.txt AVRO-1410 . Explicit version specification in pom prevents dependency management. Contributed by Eric Sammer. (tomwhite: rev 1551255) /avro/trunk/lang/java/mapred/pom.xml /avro/trunk/lang/java/pom.xml

          People

          • Assignee:
            E. Sammer
            Reporter:
            E. Sammer
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development