Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, multiple versions of the same library JAR are being pulled in for various reasons.

      • Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
      • Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.
      1. build.sh
        0.6 kB
        Tom White
      2. HADOOP-6800.patch
        4 kB
        Hemanth Yamijala
      3. HADOOP-6800.patch
        4 kB
        Tom White
      4. HADOOP-6800.patch
        3 kB
        Tom White
      5. HADOOP-6800.patch
        4 kB
        Tom White
      6. ivy-declarations.dump.merged
        8 kB
        Hemanth Yamijala
      7. jars-in-lib.dump.merged
        6 kB
        Hemanth Yamijala
      8. tar-munge
        2 kB
        Tom White

        Issue Links

          Activity

          Hide
          Tom White added a comment -

          This is related to AVRO-348, see also HADOOP-6390.

          Show
          Tom White added a comment - This is related to AVRO-348 , see also HADOOP-6390 .
          Hide
          Steve Loughran added a comment -

          +1 for this; I'd go for moving to the latest versions that didn't cause too much grief.

          HADOOP-6807 claims that SL4J is not needed for Jetty

          Show
          Steve Loughran added a comment - +1 for this; I'd go for moving to the latest versions that didn't cause too much grief. HADOOP-6807 claims that SL4J is not needed for Jetty
          Hide
          Tom White added a comment -

          This is one of three patches (others at HDFS-1212, MAPREDUCE-1870) that removes duplicate JARs. It updates a few Apache Commons JARs, JUnit, and SLF4J.

          The hardest bit was ensuring that Ant 1.6.5 and 1.7.1 don't both appear. Part of the problem here is that the Maven artifact has been relocated from the ant group to org.apache.ant. Jetty's jsp-2.1 depends on the old one, which is now explicitly excluded.

          I ran a few tests with the new configuration and they passed. I also used the script in HADOOP-6342 to test that the combined tarballs had no conflicting dependencies.

          Show
          Tom White added a comment - This is one of three patches (others at HDFS-1212 , MAPREDUCE-1870 ) that removes duplicate JARs. It updates a few Apache Commons JARs, JUnit, and SLF4J. The hardest bit was ensuring that Ant 1.6.5 and 1.7.1 don't both appear. Part of the problem here is that the Maven artifact has been relocated from the ant group to org.apache.ant. Jetty's jsp-2.1 depends on the old one, which is now explicitly excluded. I ran a few tests with the new configuration and they passed. I also used the script in HADOOP-6342 to test that the combined tarballs had no conflicting dependencies.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12447262/HADOOP-6800.patch
          against trunk revision 954647.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/587/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/587/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/587/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/587/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447262/HADOOP-6800.patch against trunk revision 954647. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/587/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/587/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/587/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/587/console This message is automatically generated.
          Hide
          Tom White added a comment -

          Is anyone able to review this (and related patches)? Thanks.

          Show
          Tom White added a comment - Is anyone able to review this (and related patches)? Thanks.
          Hide
          Tom White added a comment -

          Here's an updated version which uses Jetty's ant dependency (1.6.5) rather than paranamer's (1.7.1, which is pulled in by Avro and is not a runtime dependency).

          Show
          Tom White added a comment - Here's an updated version which uses Jetty's ant dependency (1.6.5) rather than paranamer's (1.7.1, which is pulled in by Avro and is not a runtime dependency).
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12447762/HADOOP-6800.patch
          against trunk revision 957074.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/82/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/82/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/82/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/82/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447762/HADOOP-6800.patch against trunk revision 957074. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/82/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/82/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/82/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/82/console This message is automatically generated.
          Hide
          Hemanth Yamijala added a comment -

          Tom, trying to see if I can review this, but I'm not very familiar with Ivy... Is the goal of this jira to make sure that the versions for libraries we are using in the 3 projects are the same ? I ask because there are a few libraries with different versions in the common and mapreduce projects. For e.g. commons-logging is 1.1.1 in common and 1.0.4 in mapred, and jets3t is 0.7.1 in common and 0.6.1 in mapred. These don't seem to be excluded / modified in your patch.

          Would these need to be fixed too ?

          Apologies if my understanding is completely wrong...

          Show
          Hemanth Yamijala added a comment - Tom, trying to see if I can review this, but I'm not very familiar with Ivy... Is the goal of this jira to make sure that the versions for libraries we are using in the 3 projects are the same ? I ask because there are a few libraries with different versions in the common and mapreduce projects. For e.g. commons-logging is 1.1.1 in common and 1.0.4 in mapred, and jets3t is 0.7.1 in common and 0.6.1 in mapred. These don't seem to be excluded / modified in your patch. Would these need to be fixed too ? Apologies if my understanding is completely wrong...
          Hide
          Tom White added a comment -

          Thanks for taking a look, Hemanth. Yes, the goal is to make sure that the JAR version used between the three projects are the same. My criterion for checking this was to build the three tarballs, combine them, and visually check that there were no conflicting versions in lib. This new set of patches corrects the mismatches you saw (in some cases removing the variable since its not used in the project).

          Show
          Tom White added a comment - Thanks for taking a look, Hemanth. Yes, the goal is to make sure that the JAR version used between the three projects are the same. My criterion for checking this was to build the three tarballs, combine them, and visually check that there were no conflicting versions in lib. This new set of patches corrects the mismatches you saw (in some cases removing the variable since its not used in the project).
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12447851/HADOOP-6800.patch
          against trunk revision 957074.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/593/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/593/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/593/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/593/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12447851/HADOOP-6800.patch against trunk revision 957074. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/593/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/593/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/593/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/593/console This message is automatically generated.
          Hide
          Hemanth Yamijala added a comment -

          Tom, I applied the latest patches for all three sub-projects, cleared my IVY cache, rebuilt all of them. Then, I compared the ivy library dependencies that we are declaring in the sources of the three packages and the libraries in the lib folder of the built packages. In both analyses, I still see some duplicate jars. I think most or all of the duplicates in source are coming from the HDFS Proxy ivy file. I don't know if this is reason why the libraries in the build/lib are ultimately repeating.

          For e.g. here's a list of the jars in the ivy properties that have different versions:
          commons-logging-api, commons-logging, ivy, jets3t, junit, mockito-all, slf4j-api, slf4j-log4j12

          There are quite a few repetitions in the lib folders of the three packages:
          ant,avro ,commons-http-client,commons-lang,ivy,jackson-core-asl,jackson-mapper-asl,jetty,jetty-util,junit,paranamer,paranamer-ant,paranamer-generator,qdox,servlet-api,slf4j-api,slf4j-log4j12

          I am attaching two text files that are the dumps of the ivy library declarations and the build/lib folder contents for your review.

          Please note that I did not run the cluster with these packages - so things are possibly fine with these duplications as well.

          Show
          Hemanth Yamijala added a comment - Tom, I applied the latest patches for all three sub-projects, cleared my IVY cache, rebuilt all of them. Then, I compared the ivy library dependencies that we are declaring in the sources of the three packages and the libraries in the lib folder of the built packages. In both analyses, I still see some duplicate jars. I think most or all of the duplicates in source are coming from the HDFS Proxy ivy file. I don't know if this is reason why the libraries in the build/lib are ultimately repeating. For e.g. here's a list of the jars in the ivy properties that have different versions: commons-logging-api, commons-logging, ivy, jets3t, junit, mockito-all, slf4j-api, slf4j-log4j12 There are quite a few repetitions in the lib folders of the three packages: ant,avro ,commons-http-client,commons-lang,ivy,jackson-core-asl,jackson-mapper-asl,jetty,jetty-util,junit,paranamer,paranamer-ant,paranamer-generator,qdox,servlet-api,slf4j-api,slf4j-log4j12 I am attaching two text files that are the dumps of the ivy library declarations and the build/lib folder contents for your review. Please note that I did not run the cluster with these packages - so things are possibly fine with these duplications as well.
          Hide
          Hemanth Yamijala added a comment -

          One more point is that some of the subprojects in mapred seem to be using hadoop-common-SNAPSHOT as the dependency and som hadoop-common-dev-SNAPSHOT as the dependency. However, this does not seem to be a problem as far as the libraries in the packages are concerned.

          Show
          Hemanth Yamijala added a comment - One more point is that some of the subprojects in mapred seem to be using hadoop-common- SNAPSHOT as the dependency and som hadoop-common -dev-SNAPSHOT as the dependency. However, this does not seem to be a problem as far as the libraries in the packages are concerned.
          Hide
          Tom White added a comment -

          Hemanth, Thanks for running through this. Did you run ant with "mvn-install -Dresolvers=internal" options? When I do a clean build I don't get any of the duplicates you mention.

          Show
          Tom White added a comment - Hemanth, Thanks for running through this. Did you run ant with "mvn-install -Dresolvers=internal" options? When I do a clean build I don't get any of the duplicates you mention.
          Hide
          Tom White added a comment -

          Other things I did - cleared the local maven cache, and used the "veryclean" target. The -verbose ant option is useful for seeing the dependencies ivy is pulling in.

          > One more point is that some of the subprojects in mapred seem to be using hadoop-common-SNAPSHOT as the dependency and som hadoop-common-dev-SNAPSHOT as the dependency. However, this does not seem to be a problem as far as the libraries in the packages are concerned.

          I agree this does not seem to be causing a problem here, but should be fixed in another JIRA.

          Show
          Tom White added a comment - Other things I did - cleared the local maven cache, and used the "veryclean" target. The -verbose ant option is useful for seeing the dependencies ivy is pulling in. > One more point is that some of the subprojects in mapred seem to be using hadoop-common-SNAPSHOT as the dependency and som hadoop-common-dev-SNAPSHOT as the dependency. However, this does not seem to be a problem as far as the libraries in the packages are concerned. I agree this does not seem to be causing a problem here, but should be fixed in another JIRA.
          Hide
          Hemanth Yamijala added a comment -

          Tom, I tried again:

          • Cleared Ivy cache - essentially did an rm -rf ~/.ivy2/cache
          • On common, did ant veryclean jar mvn-install – Here, I tried with and without -Dresolvers=internal
          • On hdfs, did ant veryclean jar mvn-install -Dresolvers=internal. This step fails compilation, with errors liks :
            org.apache.hadoop.hdfs.server.namenode.NameNode is not abstract and does not override abstract method refreshSuperUserGroupsConfiguration(org.apache.hadoop.conf.Configuration) in org.apache.hadoop.security.RefreshUserMappingsProtocol

          These errors do not come when I build the projects without the mvn-install option. Is there anything I'm doing incorrectly ?

          Also, please note that of the two sets of duplicates I mentioned, one set is in the ivy configuration we're defining. So, in HDFS proxy's ivy/libraries.properties, we say junit.version=3.8.2 whereas in hdfs core, we say this is 4.8.1. Similarly for other libraries mentioned in ivy-declarations.dump.merged. Shouldn't this be resolved ?

          Show
          Hemanth Yamijala added a comment - Tom, I tried again: Cleared Ivy cache - essentially did an rm -rf ~/.ivy2/cache On common, did ant veryclean jar mvn-install – Here, I tried with and without -Dresolvers=internal On hdfs, did ant veryclean jar mvn-install -Dresolvers=internal. This step fails compilation, with errors liks : org.apache.hadoop.hdfs.server.namenode.NameNode is not abstract and does not override abstract method refreshSuperUserGroupsConfiguration(org.apache.hadoop.conf.Configuration) in org.apache.hadoop.security.RefreshUserMappingsProtocol These errors do not come when I build the projects without the mvn-install option. Is there anything I'm doing incorrectly ? Also, please note that of the two sets of duplicates I mentioned, one set is in the ivy configuration we're defining. So, in HDFS proxy's ivy/libraries.properties, we say junit.version=3.8.2 whereas in hdfs core, we say this is 4.8.1. Similarly for other libraries mentioned in ivy-declarations.dump.merged. Shouldn't this be resolved ?
          Hide
          Tom White added a comment -

          Hemanth,

          I used the following command to find clashes in libraries.properties. The only problematic one was hdfsproxy's libraries.properties, as you pointed out, and which I've updated in HDFS-1212.

          find hadoop-{common,hdfs,mapreduce}-trunk/ivy hadoop-{common,hdfs,mapreduce}-trunk/src -name libraries.properties | xargs cat | sort | uniq
          

          However, even before this change, when running the attached build.sh script I got distinct libraries. (I suspect that the ones in HDFS were getting upgraded due to other dependencies.) I don't see the compile problem you had (this was with SVN revision 958322). Perhaps you can try the build.sh script with the patches from this JIRA, HDFS-1212, and MAPREDUCE-1870 applied to trunk? Thanks!

          Show
          Tom White added a comment - Hemanth, I used the following command to find clashes in libraries.properties. The only problematic one was hdfsproxy's libraries.properties, as you pointed out, and which I've updated in HDFS-1212 . find hadoop-{common,hdfs,mapreduce}-trunk/ivy hadoop-{common,hdfs,mapreduce}-trunk/src -name libraries.properties | xargs cat | sort | uniq However, even before this change, when running the attached build.sh script I got distinct libraries. (I suspect that the ones in HDFS were getting upgraded due to other dependencies.) I don't see the compile problem you had (this was with SVN revision 958322). Perhaps you can try the build.sh script with the patches from this JIRA, HDFS-1212 , and MAPREDUCE-1870 applied to trunk? Thanks!
          Hide
          Hemanth Yamijala added a comment -

          Tom, I applied the new patch on HDFS-1212. This removed all the duplicates in the ivy declarations. When I ran the project builds (using mvn-install as you suggested), I now see only 2 repetitions - ivy-2.0.0-rc2 and ivy-2.1.0 and jsp-2.1.jar and jsp-2.1-6.1.14.jar. In both the cases, both libraries were present in HDFS and mapred. So, it is not like different versions present in mapred and HDFS, but even within the project.

          I thought this was close enough and wanted to build a single distribution of the three packages and test it in pseudo-distributed mode. I was trying to use the tar-munge script, but get some errors like:
          mv: will not overwrite just-created `conf/configuration.xsl' with `hdfs/conf/configuration.xsl'

          I am still trying to resolve this, but I thought I'd update that except for the two jars I mention, everything else seems very clean.

          Show
          Hemanth Yamijala added a comment - Tom, I applied the new patch on HDFS-1212 . This removed all the duplicates in the ivy declarations. When I ran the project builds (using mvn-install as you suggested), I now see only 2 repetitions - ivy-2.0.0-rc2 and ivy-2.1.0 and jsp-2.1.jar and jsp-2.1-6.1.14.jar. In both the cases, both libraries were present in HDFS and mapred. So, it is not like different versions present in mapred and HDFS, but even within the project. I thought this was close enough and wanted to build a single distribution of the three packages and test it in pseudo-distributed mode. I was trying to use the tar-munge script, but get some errors like: mv: will not overwrite just-created `conf/configuration.xsl' with `hdfs/conf/configuration.xsl' I am still trying to resolve this, but I thought I'd update that except for the two jars I mention, everything else seems very clean.
          Hide
          Tom White added a comment -

          Hemanth, thanks for your persistence - I think we're getting close.

          Is the ivy JAR repetition because there is a copy of both in the hdfs/ivy directory? It's not a runtime dependency, so it's probably not a problem anyway.

          The JSP one is because of the directory checked into hdfs/lib, which I think can be deleted now.

          Finally, here's the latest tar-munge I've been using, which doesn't suffer the problem you describe.

          Show
          Tom White added a comment - Hemanth, thanks for your persistence - I think we're getting close. Is the ivy JAR repetition because there is a copy of both in the hdfs/ivy directory? It's not a runtime dependency, so it's probably not a problem anyway. The JSP one is because of the directory checked into hdfs/lib, which I think can be deleted now. Finally, here's the latest tar-munge I've been using, which doesn't suffer the problem you describe.
          Hide
          Hemanth Yamijala added a comment -

          Tom, I've been able to get a single node cluster up and run basic tests - hdfs and map/reduce jobs. Things appear to work fine.

          Regarding the ivy JAR, I don't see this now, so maybe there was a problem in my previous run. I think you are right about the jsp-2.1.jar. So, are you planning to delete this in this set of patches ?

          Lastly, since we've changed some of the contrib libraries, should we give a heads up - maybe on the *-dev mailing lists asking folks to take a look at the patches ?

          Show
          Hemanth Yamijala added a comment - Tom, I've been able to get a single node cluster up and run basic tests - hdfs and map/reduce jobs. Things appear to work fine. Regarding the ivy JAR, I don't see this now, so maybe there was a problem in my previous run. I think you are right about the jsp-2.1.jar. So, are you planning to delete this in this set of patches ? Lastly, since we've changed some of the contrib libraries, should we give a heads up - maybe on the *-dev mailing lists asking folks to take a look at the patches ?
          Hide
          Tom White added a comment -

          I think this is ready to go in now. I'll commit the three patches soon.

          > I think you are right about the jsp-2.1.jar. So, are you planning to delete this in this set of patches ?

          This is only in HDFS, so I will delete it there as a part of that commit.

          > Lastly, since we've changed some of the contrib libraries, should we give a heads up - maybe on the *-dev mailing lists asking folks to take a look at the patches?

          I'll send a email out.

          Show
          Tom White added a comment - I think this is ready to go in now. I'll commit the three patches soon. > I think you are right about the jsp-2.1.jar. So, are you planning to delete this in this set of patches ? This is only in HDFS, so I will delete it there as a part of that commit. > Lastly, since we've changed some of the contrib libraries, should we give a heads up - maybe on the *-dev mailing lists asking folks to take a look at the patches? I'll send a email out.
          Hide
          Hemanth Yamijala added a comment -

          Tom, about the ivy.jar, it is like you said happening because in hadoop-common there are two versions of the jar in the lib folder.

          Show
          Hemanth Yamijala added a comment - Tom, about the ivy.jar, it is like you said happening because in hadoop-common there are two versions of the jar in the lib folder.
          Hide
          Hemanth Yamijala added a comment -

          Tom, Hudson's results on MAPREDUCE-1870 seem to be OK. I checked HDFS-1212 and the test failures had age > 0; so I assume this is fine too. All patches seem ready except for reaching out to contrib devs as we discussed above. Do you need any further assistance ?

          Show
          Hemanth Yamijala added a comment - Tom, Hudson's results on MAPREDUCE-1870 seem to be OK. I checked HDFS-1212 and the test failures had age > 0; so I assume this is fine too. All patches seem ready except for reaching out to contrib devs as we discussed above. Do you need any further assistance ?
          Hide
          Hemanth Yamijala added a comment -

          Re-attaching the same patch as Tom's last one, so can run it past Hudson.

          Show
          Hemanth Yamijala added a comment - Re-attaching the same patch as Tom's last one, so can run it past Hudson.
          Hide
          Tom White added a comment -

          I've just committed this. (Hudson has already run against the patch.) Thanks for your help Hemanth.

          Show
          Tom White added a comment - I've just committed this. (Hudson has already run against the patch.) Thanks for your help Hemanth.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #317 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/317/)
          HADOOP-6800. Harmonize JAR library versions.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #317 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/317/ ) HADOOP-6800 . Harmonize JAR library versions.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #383 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/383/)
          HADOOP-6800. Harmonize JAR library versions.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #383 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/383/ ) HADOOP-6800 . Harmonize JAR library versions.

            People

            • Assignee:
              Tom White
              Reporter:
              Tom White
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development