Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1519

HDFS build is broken, ivy-resolve-common does not find hadoop-common

    Details

    • Type: Bug Bug
    • Status: Reopened
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.21.0
    • Fix Version/s: None
    • Component/s: build
    • Labels:
      None
    • Environment:

      openSUSE 11.1, Linux roisin 2.6.27.48-0.2-default #1 SMP 2010-07-29 20:06:52 +0200 x86_64 x86_64 x86_64 GNU/Linux

      Description

      HADOOP_DIR/hdfs$ ant ivy-resolve-common
      Buildfile: build.xml

      ivy-download:
      [get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
      [get] To: /usr/products/hadoop/v0_21_0/ANY/hdfs/ivy/ivy-2.1.0.jar
      [get] Not modified - so not downloaded

      ivy-init-dirs:

      ivy-probe-antlib:

      ivy-init-antlib:

      ivy-init:
      [ivy:configure] :: Ivy 2.1.0 - 20090925235825 :: http://ant.apache.org/ivy/ ::
      [ivy:configure] :: loading settings :: file = /usr/products/hadoop/v0_21_0/ANY/hdfs/ivy/ivysettings.xml

      ivy-resolve-common:
      [ivy:resolve]
      [ivy:resolve] :: problems summary ::
      [ivy:resolve] :::: WARNINGS
      [ivy:resolve] module not found: org.apache.hadoop#hadoop-common;0.21.0
      [ivy:resolve] ==== apache-snapshot: tried
      [ivy:resolve] https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.pom
      [ivy:resolve] – artifact org.apache.hadoop#hadoop-common;0.21.0!hadoop-common.jar:
      [ivy:resolve] https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.jar
      [ivy:resolve] ==== maven2: tried
      [ivy:resolve] http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.pom
      [ivy:resolve] – artifact org.apache.hadoop#hadoop-common;0.21.0!hadoop-common.jar:
      [ivy:resolve] http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.jar
      [ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
      [ivy:resolve] :: UNRESOLVED DEPENDENCIES ::
      [ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
      [ivy:resolve] :: org.apache.hadoop#hadoop-common;0.21.0: not found
      [ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
      [ivy:resolve]
      [ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

      BUILD FAILED
      /usr/products/hadoop/v0_21_0/ANY/hdfs/build.xml:1549: impossible to resolve dependencies:
      resolve failed - see output for details

      Total time: 3 seconds

        Issue Links

          Activity

          Hide
          Konstantin Boudnik added a comment -

          Usually it is an indication of:
          a) temp. service interruption of snapshop repo's server
          b) issues with local Ivy cache.

          I have cleaned local cache and ran 'ant clean compile' - build was successful:

          [ivy:resolve] downloading https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.21.0-SNAPSHOT/hadoop-common-0.21.0-20101120.093342-38.jar ...
          [ivy:resolve] ......................................................................................................................................................................................... (1259kB)
          [ivy:resolve] .. (0kB)
          [ivy:resolve]   [SUCCESSFUL ] org.apache.hadoop#hadoop-common;0.21.0-SNAPSHOT!hadoop-common.jar (3761ms)
          

          Please make sure that the issue you see isn't caused by some local reasons and re-open this bug if you still see the issue.

          Show
          Konstantin Boudnik added a comment - Usually it is an indication of: a) temp. service interruption of snapshop repo's server b) issues with local Ivy cache. I have cleaned local cache and ran 'ant clean compile' - build was successful: [ivy:resolve] downloading https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.21.0-SNAPSHOT/hadoop-common-0.21.0-20101120.093342-38.jar ... [ivy:resolve] ......................................................................................................................................................................................... (1259kB) [ivy:resolve] .. (0kB) [ivy:resolve] [SUCCESSFUL ] org.apache.hadoop#hadoop-common;0.21.0-SNAPSHOT!hadoop-common.jar (3761ms) Please make sure that the issue you see isn't caused by some local reasons and re-open this bug if you still see the issue.
          Hide
          Jon Wilson added a comment -

          $ rm -r ~/.ivy2
          HADOOP_DIR/hdfs$ ant clean compile
          – snip –
          [ivy:resolve] :: problems summary ::
          [ivy:resolve] :::: WARNINGS
          [ivy:resolve] module not found: org.apache.hadoop#hadoop-common;0.21.0
          [ivy:resolve] ==== apache-snapshot: tried
          [ivy:resolve] https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.pom
          [ivy:resolve] – artifact org.apache.hadoop#hadoop-common;0.21.0!hadoop-common.jar:
          [ivy:resolve] https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.jar
          [ivy:resolve] ==== maven2: tried
          [ivy:resolve] http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.pom
          [ivy:resolve] – artifact org.apache.hadoop#hadoop-common;0.21.0!hadoop-common.jar:
          [ivy:resolve] http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.jar
          [ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
          [ivy:resolve] :: UNRESOLVED DEPENDENCIES ::
          [ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
          [ivy:resolve] :: org.apache.hadoop#hadoop-common;0.21.0: not found
          [ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
          [ivy:resolve]
          [ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

          BUILD FAILED
          /usr/products/hadoop/v0_21_0/ANY/hdfs/build.xml:1549: impossible to resolve dependencies:
          resolve failed - see output for details

          Total time: 1 minute 45 seconds

          I still see the same problem. It looks like somehow the URL template is not correct, as the attempted URL is
          https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.jar
          rather than
          https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.21.0-SNAPSHOT/hadoop-common-0.21.0-20101120.093342-38.jar

          I've attempted to look around in the ivy xml files, but with no ivy experience, I couldn't figure out just what was constructing this URL.

          Show
          Jon Wilson added a comment - $ rm -r ~/.ivy2 HADOOP_DIR/hdfs$ ant clean compile – snip – [ivy:resolve] :: problems summary :: [ivy:resolve] :::: WARNINGS [ivy:resolve] module not found: org.apache.hadoop#hadoop-common;0.21.0 [ivy:resolve] ==== apache-snapshot: tried [ivy:resolve] https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.pom [ivy:resolve] – artifact org.apache.hadoop#hadoop-common;0.21.0!hadoop-common.jar: [ivy:resolve] https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.jar [ivy:resolve] ==== maven2: tried [ivy:resolve] http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.pom [ivy:resolve] – artifact org.apache.hadoop#hadoop-common;0.21.0!hadoop-common.jar: [ivy:resolve] http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.jar [ivy:resolve] :::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve] :: UNRESOLVED DEPENDENCIES :: [ivy:resolve] :::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve] :: org.apache.hadoop#hadoop-common;0.21.0: not found [ivy:resolve] :::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve] [ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS BUILD FAILED /usr/products/hadoop/v0_21_0/ANY/hdfs/build.xml:1549: impossible to resolve dependencies: resolve failed - see output for details Total time: 1 minute 45 seconds I still see the same problem. It looks like somehow the URL template is not correct, as the attempted URL is https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.21.0/hadoop-common-0.21.0.jar rather than https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.21.0-SNAPSHOT/hadoop-common-0.21.0-20101120.093342-38.jar I've attempted to look around in the ivy xml files, but with no ivy experience, I couldn't figure out just what was constructing this URL.
          Hide
          Konstantin Boudnik added a comment -

          What workspace do you use (e.g. what .svn/entries says at the top) and what is the revision in your workspace?

          Show
          Konstantin Boudnik added a comment - What workspace do you use (e.g. what .svn/entries says at the top) and what is the revision in your workspace?
          Hide
          Jon Wilson added a comment -

          I downloaded the tarball from http://mirror.nyi.net/apache//hadoop/core/hadoop-0.21.0/hadoop-0.21.0.tar.gz (or possibly some other mirror...), so I don't have a .svn directory.
          I unpacked the tarball into /usr/products/hadoop/v0_21_0/ANY/ (a sort of /opt type space), as user horton, and set the environment variable HADOOP_DIR to /usr/products/hadoop/v0_21_0/ANY/. I am running ant from $HADOOP_DIR/hdfs as user horton.
          When I run ant from $HADOOP_DIR/mapred, I see the same sort of problems. When I run ant from $HADOOP_DIR/common, I do not encounter any problems.

          Show
          Jon Wilson added a comment - I downloaded the tarball from http://mirror.nyi.net/apache//hadoop/core/hadoop-0.21.0/hadoop-0.21.0.tar.gz (or possibly some other mirror...), so I don't have a .svn directory. I unpacked the tarball into /usr/products/hadoop/v0_21_0/ANY/ (a sort of /opt type space), as user horton, and set the environment variable HADOOP_DIR to /usr/products/hadoop/v0_21_0/ANY/. I am running ant from $HADOOP_DIR/hdfs as user horton. When I run ant from $HADOOP_DIR/mapred, I see the same sort of problems. When I run ant from $HADOOP_DIR/common, I do not encounter any problems.
          Hide
          Konstantin Boudnik added a comment -

          The tarball you have downloaded is the release tarball. Once again, why do you need to run ant on a released version? You might have a perfectly legit reason for this, I just don't know your intentions. I am not sure if release tarballs were even suitable for running builds on them.

          If you want to build Hadoop for yourself you can take a look this link

          Show
          Konstantin Boudnik added a comment - The tarball you have downloaded is the release tarball. Once again, why do you need to run ant on a released version? You might have a perfectly legit reason for this, I just don't know your intentions. I am not sure if release tarballs were even suitable for running builds on them. If you want to build Hadoop for yourself you can take a look this link
          Hide
          Jon Wilson added a comment -

          I want to build fuse-dfs, which requires libhdfs. Do I need to get an subversion checkout to run ant instead of using a release tarball? Perhaps libhdfs and fuse-dfs are already built in the release tarball? Thanks for the help so far.

          Show
          Jon Wilson added a comment - I want to build fuse-dfs, which requires libhdfs. Do I need to get an subversion checkout to run ant instead of using a release tarball? Perhaps libhdfs and fuse-dfs are already built in the release tarball? Thanks for the help so far.
          Hide
          Konstantin Boudnik added a comment -

          I am not certain if the release tar ball has fuse-dfs included in it (check for yourself), but I am sure you should be able to build from the source code according to the a couple of simple steps in src/contrib/fuse-dfs/README

          Show
          Konstantin Boudnik added a comment - I am not certain if the release tar ball has fuse-dfs included in it (check for yourself), but I am sure you should be able to build from the source code according to the a couple of simple steps in src/contrib/fuse-dfs/README
          Hide
          Jon Wilson added a comment -

          I've been following the instructions at http://wiki.apache.org/hadoop/MountableHDFS, which worked (with minor modifications) with the release tarball for 0.20.2. However, my application (ROOT: http://root.cern.ch) opens files with mode O_RDWR (it doesn't do anything that hdfs doesn't support with them, though), so I needed the later version with HDFS-861 fixed.

          I did a bit of poking, and it looks like all the code for fuse-dfs and for libhdfs is present, but that libhdfs is not built in the release tarball. So I do need to build libhdfs. Is ant the recommended way to do this? Looking at http://wiki.apache.org/hadoop/LibHDFS, perhaps I should just use make instead. I'll try that, and report back.

          Show
          Jon Wilson added a comment - I've been following the instructions at http://wiki.apache.org/hadoop/MountableHDFS , which worked (with minor modifications) with the release tarball for 0.20.2. However, my application (ROOT: http://root.cern.ch ) opens files with mode O_RDWR (it doesn't do anything that hdfs doesn't support with them, though), so I needed the later version with HDFS-861 fixed. I did a bit of poking, and it looks like all the code for fuse-dfs and for libhdfs is present, but that libhdfs is not built in the release tarball. So I do need to build libhdfs. Is ant the recommended way to do this? Looking at http://wiki.apache.org/hadoop/LibHDFS , perhaps I should just use make instead. I'll try that, and report back.
          Hide
          Konstantin Boudnik added a comment -

          It isn't like ant is the recommended way to build libhdfs. It might be simpler to do though. Basically all you need (if you have source tree handy) is to run ant compile-contrib -Dcompile.c+=yes -Dlibhdfs=yes and after a short while you'll find the library binaries under build/c+

          Show
          Konstantin Boudnik added a comment - It isn't like ant is the recommended way to build libhdfs. It might be simpler to do though. Basically all you need (if you have source tree handy) is to run ant compile-contrib -Dcompile.c+ =yes -Dlibhdfs=yes and after a short while you'll find the library binaries under build/c +
          Hide
          Jon Wilson added a comment -

          I did find that both libhdfs and fuse-dfs are built in the release tarball. Luckily, they do appear to be built for the correct architecture for me, so things are well for me. My issues are resolved by this, but perhaps the JIRA issue should remain open, since ant still cannot be run on a release tarball.

          The instructions on the wiki at MountableHDFS and libHDFS should be updated to note that these are built in the release tarball, or ant in the release should be fixed (or more likely both).

          Show
          Jon Wilson added a comment - I did find that both libhdfs and fuse-dfs are built in the release tarball. Luckily, they do appear to be built for the correct architecture for me, so things are well for me. My issues are resolved by this, but perhaps the JIRA issue should remain open, since ant still cannot be run on a release tarball. The instructions on the wiki at MountableHDFS and libHDFS should be updated to note that these are built in the release tarball, or ant in the release should be fixed (or more likely both).
          Hide
          Konstantin Boudnik added a comment -

          Tom White (the release manager for 0.21) has pointed out that post-split releases aren't easy to build together (exactly because they are split in three different parts).

          However, this dependency issue seems like something else however related to the overall problem.

          Show
          Konstantin Boudnik added a comment - Tom White (the release manager for 0.21) has pointed out that post-split releases aren't easy to build together (exactly because they are split in three different parts). However, this dependency issue seems like something else however related to the overall problem.
          Hide
          Tony Valderrama added a comment -

          I was building thriftfs, and got all of hdfs to rebuild with the following modifications. This was mostly just bumping around in config files until it worked, so somebody who knows the build system should make an actual patch. I also had to manually run 'mkdir hdfs/lib' because ant was complaining that it didn't exist.

          hdfs/ivy.xml
          ---     <dependency org="org.apache.hadoop" name="hadoop-common" rev="${hadoop-common.version}" conf="common->default"/>
          +++     <dependency org="org.apache.hadoop" name="hadoop-common" rev="${hadoop-common.version}-SNAPSHOT" conf="common->default"/>
          
          hdfs/src/contrib/hdfsproxy/ivy.xml
                 <dependency org="org.apache.hadoop"
                   name="hadoop-common"
          ---      rev="${hadoop-common.version}"
          +++      rev="${hadoop-common.version}-SNAPSHOT"
                   conf="common->default"/>
                 <dependency org="org.apache.hadoop"
                   name="hadoop-common-test"
          ---      rev="${hadoop-common.version}"
          +++      rev="${hadoop-common.version}-SNAPSHOT"
                   conf="common->default"/>
          
          hdfs/src/contrib/thriftfs/ivy.xml
                 <dependency org="org.apache.hadoop"
                   name="hadoop-common"
          ---      rev="${hadoop-common.version}"
          +++      rev="${hadoop-common.version}-SNAPSHOT"
                   conf="common->default"/>
          
          Show
          Tony Valderrama added a comment - I was building thriftfs, and got all of hdfs to rebuild with the following modifications. This was mostly just bumping around in config files until it worked, so somebody who knows the build system should make an actual patch. I also had to manually run 'mkdir hdfs/lib' because ant was complaining that it didn't exist. hdfs/ivy.xml --- <dependency org="org.apache.hadoop" name="hadoop-common" rev="${hadoop-common.version}" conf="common->default"/> +++ <dependency org="org.apache.hadoop" name="hadoop-common" rev="${hadoop-common.version}-SNAPSHOT" conf="common->default"/> hdfs/src/contrib/hdfsproxy/ivy.xml <dependency org="org.apache.hadoop" name="hadoop-common" --- rev="${hadoop-common.version}" +++ rev="${hadoop-common.version}-SNAPSHOT" conf="common->default"/> <dependency org="org.apache.hadoop" name="hadoop-common-test" --- rev="${hadoop-common.version}" +++ rev="${hadoop-common.version}-SNAPSHOT" conf="common->default"/> hdfs/src/contrib/thriftfs/ivy.xml <dependency org="org.apache.hadoop" name="hadoop-common" --- rev="${hadoop-common.version}" +++ rev="${hadoop-common.version}-SNAPSHOT" conf="common->default"/>
          Hide
          Yanbo Liang added a comment -

          I was building hdfs checked out from svn and met the same problem.
          Is it the repository has not been updated?
          How to resolve this problem?

          Show
          Yanbo Liang added a comment - I was building hdfs checked out from svn and met the same problem. Is it the repository has not been updated? How to resolve this problem?
          Hide
          Wei Lu added a comment -

          @Tony Valderrama
          +1 I agree. I had to do the same. And your config changes worked for me.

          Show
          Wei Lu added a comment - @Tony Valderrama +1 I agree. I had to do the same. And your config changes worked for me.

            People

            • Assignee:
              Unassigned
              Reporter:
              Jon Wilson
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:

                Development