Hadoop Common
  1. Hadoop Common
  2. HADOOP-6370

Contrib project ivy dependencies are not included in binary target

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: build
    • Labels:
      None

      Description

      Only Hadoop's own library dependencies are promoted to $

      {build.dir}

      /lib; any libraries required by contribs are not redistributed.

      1. HADOOP-6370.patch
        0.5 kB
        Aaron Kimball
      2. HADOOP-6370.2.patch
        4 kB
        Aaron Kimball

        Issue Links

          Activity

          Hide
          Eli Collins added a comment -

          Can we close this now that trunk has been mavenized or do people still want this for 1.x?

          Show
          Eli Collins added a comment - Can we close this now that trunk has been mavenized or do people still want this for 1.x?
          Tom White made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Tom White added a comment -

          Unfortunately this has fallen out of date. Aaron, would you like to regenerate it? Thanks.

          Show
          Tom White added a comment - Unfortunately this has fallen out of date. Aaron, would you like to regenerate it? Thanks.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12429616/HADOOP-6370.2.patch
          against trunk revision 1071364.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/282//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429616/HADOOP-6370.2.patch against trunk revision 1071364. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/282//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12429616/HADOOP-6370.2.patch
          against trunk revision 1031422.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/67//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429616/HADOOP-6370.2.patch against trunk revision 1031422. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/67//console This message is automatically generated.
          Hide
          Aaron Kimball added a comment -

          Steve, I think that only applies to conflicts within a given module's (transitive) dependency set. Looking at the ivy docs, I can't see a way to do multiple ivy:resolve calls in the same build. Performing a resolve will set some ant build properties and attempting to override these (e.g., by doing a base-hadoop resolve and then a contrib module's resolve) will fail. There's also no way to specify multiple ivy.xml files, nor is there a way to look at the artifact jars that (may) already exist in another directory (e.g., the ivy cache for the base-hadoop project).

          I think that eventually auto-detecting these problems is a good goal, but I think for now the status quo of "contrib owners are responsible for dealing with their own version conflicts" isn't going to change. I don't see a straightforward technical means of dealing with this.

          Show
          Aaron Kimball added a comment - Steve, I think that only applies to conflicts within a given module's (transitive) dependency set. Looking at the ivy docs, I can't see a way to do multiple ivy:resolve calls in the same build. Performing a resolve will set some ant build properties and attempting to override these (e.g., by doing a base-hadoop resolve and then a contrib module's resolve) will fail. There's also no way to specify multiple ivy.xml files, nor is there a way to look at the artifact jars that (may) already exist in another directory (e.g., the ivy cache for the base-hadoop project). I think that eventually auto-detecting these problems is a good goal, but I think for now the status quo of "contrib owners are responsible for dealing with their own version conflicts" isn't going to change. I don't see a straightforward technical means of dealing with this.
          Hide
          steve_l added a comment -

          There is some way you can tell ivy to resolve a set of dependencies and fail if there are conflicts, which could be used to check for version compatibilities. I don't know how the dependencies from external artifacts would affect this.

          Show
          steve_l added a comment - There is some way you can tell ivy to resolve a set of dependencies and fail if there are conflicts, which could be used to check for version compatibilities. I don't know how the dependencies from external artifacts would affect this.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12429616/HADOOP-6370.2.patch
          against trunk revision 896691.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/264/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/264/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/264/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/264/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429616/HADOOP-6370.2.patch against trunk revision 896691. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/264/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/264/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/264/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/264/console This message is automatically generated.
          Aaron Kimball made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Aaron Kimball made changes -
          Attachment HADOOP-6370.2.patch [ 12429616 ]
          Hide
          Aaron Kimball added a comment -

          new patch for this issue. I discovered in the process that 'ant package' on the toplevel does not invoke package targets in the contribs. This is now fixed as well. Several contribs have no-op package targets; their authors may want to turn some attention to them, but that's a separate issue.

          Show
          Aaron Kimball added a comment - new patch for this issue. I discovered in the process that 'ant package' on the toplevel does not invoke package targets in the contribs. This is now fixed as well. Several contribs have no-op package targets; their authors may want to turn some attention to them, but that's a separate issue.
          Hide
          Tom White added a comment -

          Not sure there is anything extra to do with the classpath, since users wishing to use a particular contrib module can either copy all the jars to the top-level lib, or set HADOOP_CLASSPATH to include the extra jars.

          Show
          Tom White added a comment - Not sure there is anything extra to do with the classpath, since users wishing to use a particular contrib module can either copy all the jars to the top-level lib, or set HADOOP_CLASSPATH to include the extra jars.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          To put a particular contrib module's dependencies into contrib/<project>/lib we could copy all of its dependencies there, then remove duplicates using a present selector (http://ant.apache.org/manual/CoreTypes/selectors.html#presentselect).

          I like this idea. +1. We should also ensure that classpaths of the contrib projects are set accordingly.

          Show
          Vinod Kumar Vavilapalli added a comment - To put a particular contrib module's dependencies into contrib/<project>/lib we could copy all of its dependencies there, then remove duplicates using a present selector ( http://ant.apache.org/manual/CoreTypes/selectors.html#presentselect ). I like this idea. +1. We should also ensure that classpaths of the contrib projects are set accordingly.
          Hide
          Aaron Kimball added a comment -

          That seems reasonable.

          Show
          Aaron Kimball added a comment - That seems reasonable.
          Tom White made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Tom White added a comment -

          It looks like the contrib dependencies have never been distributed (e.g. index depends on Lucene, but the Lucene jar has not been bundled). I think it probably makes sense to distribute the dependencies though.

          To put a particular contrib module's dependencies into contrib/<project>/lib we could copy all of its dependencies there, then remove duplicates using a present selector (http://ant.apache.org/manual/CoreTypes/selectors.html#presentselect). Could this work?

          Show
          Tom White added a comment - It looks like the contrib dependencies have never been distributed (e.g. index depends on Lucene, but the Lucene jar has not been bundled). I think it probably makes sense to distribute the dependencies though. To put a particular contrib module's dependencies into contrib/<project>/lib we could copy all of its dependencies there, then remove duplicates using a present selector ( http://ant.apache.org/manual/CoreTypes/selectors.html#presentselect ). Could this work?
          Hide
          Tom White added a comment -

          Do you know of a straightforward way to do this?

          No, I don't. If we put the contrib dependency jars into the top-level lib then we'd have to be sure there aren't version inconsistencies, which has happened before (see HADOOP-6395).

          Show
          Tom White added a comment - Do you know of a straightforward way to do this? No, I don't. If we put the contrib dependency jars into the top-level lib then we'd have to be sure there aren't version inconsistencies, which has happened before (see HADOOP-6395 ).
          Hide
          Aaron Kimball added a comment -

          Possibly. It's worth noting that contrib projects also pull in many of the same jars as those used by Hadoop itself. So we'd want to deduplicate from those used by the common project first, or else we'll wind up having a very bloated binary tarball. Do you know of a straightforward way to do this?

          Show
          Aaron Kimball added a comment - Possibly. It's worth noting that contrib projects also pull in many of the same jars as those used by Hadoop itself. So we'd want to deduplicate from those used by the common project first, or else we'll wind up having a very bloated binary tarball. Do you know of a straightforward way to do this?
          Hide
          Tom White added a comment -

          I wonder if contrib-specific dependencies should go into contrib/<project>/lib. The contrib jar files are not on the classpath by default, and putting their dependencies in a separate directory would be equivalent.

          Show
          Tom White added a comment - I wonder if contrib-specific dependencies should go into contrib/<project>/lib. The contrib jar files are not on the classpath by default, and putting their dependencies in a separate directory would be equivalent.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12424764/HADOOP-6370.patch
          against trunk revision 833553.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/135/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/135/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/135/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/135/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12424764/HADOOP-6370.patch against trunk revision 833553. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/135/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/135/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/135/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/135/console This message is automatically generated.
          Aaron Kimball made changes -
          Link This issue relates to HDFS-768 [ HDFS-768 ]
          Aaron Kimball made changes -
          Link This issue relates to MAPREDUCE-1212 [ MAPREDUCE-1212 ]
          Aaron Kimball made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Aaron Kimball made changes -
          Field Original Value New Value
          Attachment HADOOP-6370.patch [ 12424764 ]
          Hide
          Aaron Kimball added a comment -

          Attaching a patch which corrects the issue in common. Tested by using the makefile from HADOOP-6342 to combine all three projects, started a pseudo-distributed cluster, ran some example jobs in it.

          Show
          Aaron Kimball added a comment - Attaching a patch which corrects the issue in common. Tested by using the makefile from HADOOP-6342 to combine all three projects, started a pseudo-distributed cluster, ran some example jobs in it.
          Aaron Kimball created issue -

            People

            • Assignee:
              Aaron Kimball
              Reporter:
              Aaron Kimball
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development