Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2189

guava-r09 dependency missing from "ivy/hadoop-hdfs-template.xml" in HDFS.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      1. patch.txt
        0.5 kB
        Plamen Jeliazkov
      2. HDFS-2189-1.patch
        0.5 kB
        Joep Rottinghuis

        Issue Links

          Activity

          Hide
          Joep Rottinghuis added a comment -

          I opened a separate bug for the build issue, which is related to the version of Ant.
          See: HDFS-2315

          Show
          Joep Rottinghuis added a comment - I opened a separate bug for the build issue, which is related to the version of Ant. See: HDFS-2315
          Hide
          Joep Rottinghuis added a comment -

          The Hadoop-Hdfs-22-branch build on Jenkins fails with the following:
          BUILD FAILED
          /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-22-branch/trunk/build.xml:1288: The following error occurred while executing this line:
          /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-22-branch/trunk/src/contrib/build.xml:60: The following error occurred while executing this line:
          /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-22-branch/trunk/src/contrib/fuse-dfs/build.xml:22: The following error occurred while executing this line:
          /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-22-branch/trunk/src/contrib/build-contrib.xml:68: Source resource does not exist: /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-22-branch/trunk/src/contrib/ivy/libraries.properties

          When I revert the version of Ant on my desktop from 1.8.1 to 1.6.5 I see the same issue.
          However, it seems that HDFS does not build with 1.6.5 anyway:
          BUILD FAILED
          /home/jrottinghuis/git/hadoop-common/hdfs/build.xml:1545: Class org.apache.tools.ant.taskdefs.ConditionTask doesn't support the nested "typefound" element.
          This is supposed to be fixed with ant 1.7.0.

          I suspect with the Jenkins slaves failing, there is now a different version of ant on the current slave.
          What version of ant is supposed to be used on the apache Jenkins servers?
          That way I can install the same version and make sure the builds go through.

          Show
          Joep Rottinghuis added a comment - The Hadoop-Hdfs-22-branch build on Jenkins fails with the following: BUILD FAILED /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-22-branch/trunk/build.xml:1288: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-22-branch/trunk/src/contrib/build.xml:60: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-22-branch/trunk/src/contrib/fuse-dfs/build.xml:22: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-22-branch/trunk/src/contrib/build-contrib.xml:68: Source resource does not exist: /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-22-branch/trunk/src/contrib/ivy/libraries.properties When I revert the version of Ant on my desktop from 1.8.1 to 1.6.5 I see the same issue. However, it seems that HDFS does not build with 1.6.5 anyway: BUILD FAILED /home/jrottinghuis/git/hadoop-common/hdfs/build.xml:1545: Class org.apache.tools.ant.taskdefs.ConditionTask doesn't support the nested "typefound" element. This is supposed to be fixed with ant 1.7.0. I suspect with the Jenkins slaves failing, there is now a different version of ant on the current slave. What version of ant is supposed to be used on the apache Jenkins servers? That way I can install the same version and make sure the builds go through.
          Hide
          Joep Rottinghuis added a comment -

          Code has been fixed but mvn:publish target must be run to publish jar+pom. Downstream builds are failing.

          Show
          Joep Rottinghuis added a comment - Code has been fixed but mvn:publish target must be run to publish jar+pom. Downstream builds are failing.
          Hide
          Joep Rottinghuis added a comment -

          Integration build should be kicked because the last published POM still have the erroneous reference to org.apache.hadooip#guava. This is failing downstream builds. See HBASE-4327

          Show
          Joep Rottinghuis added a comment - Integration build should be kicked because the last published POM still have the erroneous reference to org.apache.hadooip#guava. This is failing downstream builds. See HBASE-4327
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-22-branch #73 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/73/)
          HDFS-2189. Recording issue in CHANGES.txt
          HDFS-2189. Add guava-r09 dependency to ivy/hadoop-hdfs-template.xml. Contributed by Joep Rottinghuis
          Revert HDFS-2189.

          shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153695
          Files :

          • /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt

          shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153659
          Files :

          • /hadoop/common/branches/branch-0.22/hdfs/ivy/hadoop-hdfs-template.xml

          shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152384
          Files :

          • /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.22/hdfs/ivy/hadoop-hdfs-template.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #73 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/73/ ) HDFS-2189 . Recording issue in CHANGES.txt HDFS-2189 . Add guava-r09 dependency to ivy/hadoop-hdfs-template.xml. Contributed by Joep Rottinghuis Revert HDFS-2189 . shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153695 Files : /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1153659 Files : /hadoop/common/branches/branch-0.22/hdfs/ivy/hadoop-hdfs-template.xml shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152384 Files : /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt /hadoop/common/branches/branch-0.22/hdfs/ivy/hadoop-hdfs-template.xml
          Hide
          Konstantin Shvachko added a comment -

          I just committed the new patch. Thank you Joep and Plamen.

          Show
          Konstantin Shvachko added a comment - I just committed the new patch. Thank you Joep and Plamen.
          Hide
          Joep Rottinghuis added a comment -

          Attaching new patch (2189-1).
          Indeed hdfs depends on guava.
          Specificaly o.a.h.hdfs.SocketCache imports the com.google. classes.
          Note that I generated this patch after applying the HDFS-2214 patch first.

          Show
          Joep Rottinghuis added a comment - Attaching new patch (2189-1). Indeed hdfs depends on guava. Specificaly o.a.h.hdfs.SocketCache imports the com.google. classes. Note that I generated this patch after applying the HDFS-2214 patch first.
          Hide
          Konstantin Shvachko added a comment -

          I reverted the patch based on that.

          Show
          Konstantin Shvachko added a comment - I reverted the patch based on that.
          Hide
          Joep Rottinghuis added a comment -

          I was asked why this patch has to be rolled back.
          This patch adds a dependency on org.apache.hadoop#guava version r09.
          The modified file is used to generate a POM.
          In downstream builds this POM is used to determine what other dependencies need to be pulled in when the hadoop-hdfs jar is pulled in.
          Problem is that org.apache.hadoop#guava does not exist, it anything it should be com.google.guava#guava.
          In other words the group ID is wrong.

          Moreover, it is not clear this is the correct fix for the original mapreduce compilation error in the first place. It is likely that the proper fix is to put the dependency in the template for mapreduce (resulting in the guava jar to be downloaded in the lib directory for the mapreduce build). Preliminary tests show this to be the case, but I need to do further testing.

          One way or the other, this patch does not work and causes downstream problems.

          Show
          Joep Rottinghuis added a comment - I was asked why this patch has to be rolled back. This patch adds a dependency on org.apache.hadoop#guava version r09. The modified file is used to generate a POM. In downstream builds this POM is used to determine what other dependencies need to be pulled in when the hadoop-hdfs jar is pulled in. Problem is that org.apache.hadoop#guava does not exist, it anything it should be com.google.guava#guava. In other words the group ID is wrong. Moreover, it is not clear this is the correct fix for the original mapreduce compilation error in the first place. It is likely that the proper fix is to put the dependency in the template for mapreduce (resulting in the guava jar to be downloaded in the lib directory for the mapreduce build). Preliminary tests show this to be the case, but I need to do further testing. One way or the other, this patch does not work and causes downstream problems.
          Hide
          Joep Rottinghuis added a comment -

          After Apache pushes a new 0.22.0-SNAPSHOT for HDFS jars to the Maven repo, you may be to clear out the local maven repo (/.m2) and ivy cache (/.ivy2) to get downstream builds working properly again.

          Show
          Joep Rottinghuis added a comment - After Apache pushes a new 0.22.0-SNAPSHOT for HDFS jars to the Maven repo, you may be to clear out the local maven repo ( /.m2) and ivy cache ( /.ivy2) to get downstream builds working properly again.
          Hide
          Joep Rottinghuis added a comment -

          I need to do a little more work to determine if the depency should really be in the template of mapreduce, or in hdfs.
          In either case this patch needs to be reverted because it has the wrong group ID:org.apache.hadoop

          Downstream buids such as mapreduce will fail with an error that org.apache.hadoop#guava cannote be resolved.
          The correct group ID should be com.google.guava.

          Show
          Joep Rottinghuis added a comment - I need to do a little more work to determine if the depency should really be in the template of mapreduce, or in hdfs. In either case this patch needs to be reverted because it has the wrong group ID:org.apache.hadoop Downstream buids such as mapreduce will fail with an error that org.apache.hadoop#guava cannote be resolved. The correct group ID should be com.google.guava.
          Hide
          Plamen Jeliazkov added a comment -

          This patch needs to be reverted. After doing a new build we noticed this patch did not resolve anything.

          The real fix lies within MAPREDUCE's template XML file; will be resolved later.

          Show
          Plamen Jeliazkov added a comment - This patch needs to be reverted. After doing a new build we noticed this patch did not resolve anything. The real fix lies within MAPREDUCE's template XML file; will be resolved later.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-22-branch #72 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/72/)
          HDFS-2189. Add guava-r09 dependency to ivy/hadoop-hdfs-template.xml. Contributed by Plamen Jeliazkov.

          shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1150967
          Files :

          • /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.22/hdfs/ivy/hadoop-hdfs-template.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #72 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/72/ ) HDFS-2189 . Add guava-r09 dependency to ivy/hadoop-hdfs-template.xml. Contributed by Plamen Jeliazkov. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1150967 Files : /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt /hadoop/common/branches/branch-0.22/hdfs/ivy/hadoop-hdfs-template.xml
          Hide
          Konstantin Shvachko added a comment -

          I just committed this.
          Thank you Plamen.

          Show
          Konstantin Shvachko added a comment - I just committed this. Thank you Plamen.
          Hide
          Konstantin Shvachko added a comment -

          +1 on the patch.

          Show
          Konstantin Shvachko added a comment - +1 on the patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12487578/patch.txt
          against trunk revision 1149771.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.TestHDFSTrash

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1000//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1000//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1000//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12487578/patch.txt against trunk revision 1149771. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestHDFSTrash +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1000//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1000//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1000//console This message is automatically generated.

            People

            • Assignee:
              Joep Rottinghuis
              Reporter:
              Plamen Jeliazkov
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development