Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2189

guava-r09 dependency missing from "ivy/hadoop-hdfs-template.xml" in HDFS.

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.22.0
    • 0.22.0
    • None
    • None
    • Reviewed

    Description

      Attachments

        1. patch.txt
          0.5 kB
          Plamen Jeliazkov
        2. HDFS-2189-1.patch
          0.5 kB
          Joep Rottinghuis

        Issue Links

          Activity

            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12487578/patch.txt
            against trunk revision 1149771.

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no new tests are needed for this patch.
            Also please list what manual steps were performed to verify this patch.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            -1 core tests. The patch failed these core unit tests:
            org.apache.hadoop.hdfs.TestHDFSTrash

            +1 contrib tests. The patch passed contrib unit tests.

            +1 system test framework. The patch passed system test framework compile.

            Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1000//testReport/
            Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1000//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1000//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12487578/patch.txt against trunk revision 1149771. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestHDFSTrash +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1000//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1000//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1000//console This message is automatically generated.

            +1 on the patch.

            shv Konstantin Shvachko added a comment - +1 on the patch.

            I just committed this.
            Thank you Plamen.

            shv Konstantin Shvachko added a comment - I just committed this. Thank you Plamen.
            hudson Hudson added a comment -

            Integrated in Hadoop-Hdfs-22-branch #72 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/72/)
            HDFS-2189. Add guava-r09 dependency to ivy/hadoop-hdfs-template.xml. Contributed by Plamen Jeliazkov.

            shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1150967
            Files :

            • /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt
            • /hadoop/common/branches/branch-0.22/hdfs/ivy/hadoop-hdfs-template.xml
            hudson Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #72 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/72/ ) HDFS-2189 . Add guava-r09 dependency to ivy/hadoop-hdfs-template.xml. Contributed by Plamen Jeliazkov. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1150967 Files : /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt /hadoop/common/branches/branch-0.22/hdfs/ivy/hadoop-hdfs-template.xml

            This patch needs to be reverted. After doing a new build we noticed this patch did not resolve anything.

            The real fix lies within MAPREDUCE's template XML file; will be resolved later.

            zero45 Plamen Jeliazkov added a comment - This patch needs to be reverted. After doing a new build we noticed this patch did not resolve anything. The real fix lies within MAPREDUCE's template XML file; will be resolved later.

            I need to do a little more work to determine if the depency should really be in the template of mapreduce, or in hdfs.
            In either case this patch needs to be reverted because it has the wrong group ID:org.apache.hadoop

            Downstream buids such as mapreduce will fail with an error that org.apache.hadoop#guava cannote be resolved.
            The correct group ID should be com.google.guava.

            jrottinghuis Joep Rottinghuis added a comment - I need to do a little more work to determine if the depency should really be in the template of mapreduce, or in hdfs. In either case this patch needs to be reverted because it has the wrong group ID:org.apache.hadoop Downstream buids such as mapreduce will fail with an error that org.apache.hadoop#guava cannote be resolved. The correct group ID should be com.google.guava.

            After Apache pushes a new 0.22.0-SNAPSHOT for HDFS jars to the Maven repo, you may be to clear out the local maven repo (/.m2) and ivy cache (/.ivy2) to get downstream builds working properly again.

            jrottinghuis Joep Rottinghuis added a comment - After Apache pushes a new 0.22.0-SNAPSHOT for HDFS jars to the Maven repo, you may be to clear out the local maven repo ( /.m2) and ivy cache ( /.ivy2) to get downstream builds working properly again.

            I was asked why this patch has to be rolled back.
            This patch adds a dependency on org.apache.hadoop#guava version r09.
            The modified file is used to generate a POM.
            In downstream builds this POM is used to determine what other dependencies need to be pulled in when the hadoop-hdfs jar is pulled in.
            Problem is that org.apache.hadoop#guava does not exist, it anything it should be com.google.guava#guava.
            In other words the group ID is wrong.

            Moreover, it is not clear this is the correct fix for the original mapreduce compilation error in the first place. It is likely that the proper fix is to put the dependency in the template for mapreduce (resulting in the guava jar to be downloaded in the lib directory for the mapreduce build). Preliminary tests show this to be the case, but I need to do further testing.

            One way or the other, this patch does not work and causes downstream problems.

            jrottinghuis Joep Rottinghuis added a comment - I was asked why this patch has to be rolled back. This patch adds a dependency on org.apache.hadoop#guava version r09. The modified file is used to generate a POM. In downstream builds this POM is used to determine what other dependencies need to be pulled in when the hadoop-hdfs jar is pulled in. Problem is that org.apache.hadoop#guava does not exist, it anything it should be com.google.guava#guava. In other words the group ID is wrong. Moreover, it is not clear this is the correct fix for the original mapreduce compilation error in the first place. It is likely that the proper fix is to put the dependency in the template for mapreduce (resulting in the guava jar to be downloaded in the lib directory for the mapreduce build). Preliminary tests show this to be the case, but I need to do further testing. One way or the other, this patch does not work and causes downstream problems.

            I reverted the patch based on that.

            shv Konstantin Shvachko added a comment - I reverted the patch based on that.

            Attaching new patch (2189-1).
            Indeed hdfs depends on guava.
            Specificaly o.a.h.hdfs.SocketCache imports the com.google. classes.
            Note that I generated this patch after applying the HDFS-2214 patch first.

            jrottinghuis Joep Rottinghuis added a comment - Attaching new patch (2189-1). Indeed hdfs depends on guava. Specificaly o.a.h.hdfs.SocketCache imports the com.google. classes. Note that I generated this patch after applying the HDFS-2214 patch first.

            I just committed the new patch. Thank you Joep and Plamen.

            shv Konstantin Shvachko added a comment - I just committed the new patch. Thank you Joep and Plamen.
            hudson Hudson added a comment -

            Integrated in Hadoop-Hdfs-22-branch #73 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/73/)
            HDFS-2189. Recording issue in CHANGES.txt
            HDFS-2189. Add guava-r09 dependency to ivy/hadoop-hdfs-template.xml. Contributed by Joep Rottinghuis
            Revert HDFS-2189.

            shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153695
            Files :

            • /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt

            shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153659
            Files :

            • /hadoop/common/branches/branch-0.22/hdfs/ivy/hadoop-hdfs-template.xml

            shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152384
            Files :

            • /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt
            • /hadoop/common/branches/branch-0.22/hdfs/ivy/hadoop-hdfs-template.xml
            hudson Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #73 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/73/ ) HDFS-2189 . Recording issue in CHANGES.txt HDFS-2189 . Add guava-r09 dependency to ivy/hadoop-hdfs-template.xml. Contributed by Joep Rottinghuis Revert HDFS-2189 . shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153695 Files : /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153659 Files : /hadoop/common/branches/branch-0.22/hdfs/ivy/hadoop-hdfs-template.xml shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152384 Files : /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt /hadoop/common/branches/branch-0.22/hdfs/ivy/hadoop-hdfs-template.xml

            Integration build should be kicked because the last published POM still have the erroneous reference to org.apache.hadooip#guava. This is failing downstream builds. See HBASE-4327

            jrottinghuis Joep Rottinghuis added a comment - Integration build should be kicked because the last published POM still have the erroneous reference to org.apache.hadooip#guava. This is failing downstream builds. See HBASE-4327

            Code has been fixed but mvn:publish target must be run to publish jar+pom. Downstream builds are failing.

            jrottinghuis Joep Rottinghuis added a comment - Code has been fixed but mvn:publish target must be run to publish jar+pom. Downstream builds are failing.

            The Hadoop-Hdfs-22-branch build on Jenkins fails with the following:
            BUILD FAILED
            /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-22-branch/trunk/build.xml:1288: The following error occurred while executing this line:
            /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-22-branch/trunk/src/contrib/build.xml:60: The following error occurred while executing this line:
            /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-22-branch/trunk/src/contrib/fuse-dfs/build.xml:22: The following error occurred while executing this line:
            /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-22-branch/trunk/src/contrib/build-contrib.xml:68: Source resource does not exist: /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-22-branch/trunk/src/contrib/ivy/libraries.properties

            When I revert the version of Ant on my desktop from 1.8.1 to 1.6.5 I see the same issue.
            However, it seems that HDFS does not build with 1.6.5 anyway:
            BUILD FAILED
            /home/jrottinghuis/git/hadoop-common/hdfs/build.xml:1545: Class org.apache.tools.ant.taskdefs.ConditionTask doesn't support the nested "typefound" element.
            This is supposed to be fixed with ant 1.7.0.

            I suspect with the Jenkins slaves failing, there is now a different version of ant on the current slave.
            What version of ant is supposed to be used on the apache Jenkins servers?
            That way I can install the same version and make sure the builds go through.

            jrottinghuis Joep Rottinghuis added a comment - The Hadoop-Hdfs-22-branch build on Jenkins fails with the following: BUILD FAILED /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-22-branch/trunk/build.xml:1288: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-22-branch/trunk/src/contrib/build.xml:60: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-22-branch/trunk/src/contrib/fuse-dfs/build.xml:22: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-22-branch/trunk/src/contrib/build-contrib.xml:68: Source resource does not exist: /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-22-branch/trunk/src/contrib/ivy/libraries.properties When I revert the version of Ant on my desktop from 1.8.1 to 1.6.5 I see the same issue. However, it seems that HDFS does not build with 1.6.5 anyway: BUILD FAILED /home/jrottinghuis/git/hadoop-common/hdfs/build.xml:1545: Class org.apache.tools.ant.taskdefs.ConditionTask doesn't support the nested "typefound" element. This is supposed to be fixed with ant 1.7.0. I suspect with the Jenkins slaves failing, there is now a different version of ant on the current slave. What version of ant is supposed to be used on the apache Jenkins servers? That way I can install the same version and make sure the builds go through.

            I opened a separate bug for the build issue, which is related to the version of Ant.
            See: HDFS-2315

            jrottinghuis Joep Rottinghuis added a comment - I opened a separate bug for the build issue, which is related to the version of Ant. See: HDFS-2315

            People

              jrottinghuis Joep Rottinghuis
              zero45 Plamen Jeliazkov
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: