Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1935

Build should not redownload ivy on every invocation

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: build
    • Labels:

      Description

      Currently we re-download ivy every time we build. If the jar already exists, we should skip this.

      1. diff
        0.8 kB
        Plamen Jeliazkov
      2. hdfs-1935.patch
        0.8 kB
        Plamen Jeliazkov
      3. hdfs-1935.txt
        2 kB
        Todd Lipcon

        Issue Links

          Activity

          Hide
          Todd Lipcon added a comment -

          Trivial build fix. The only issue with this is that it depends on ant 1.8.0.

          Are we OK with making this a dependency? It's been released for over a year.

          Show
          Todd Lipcon added a comment - Trivial build fix. The only issue with this is that it depends on ant 1.8.0. Are we OK with making this a dependency? It's been released for over a year.
          Hide
          Jolly Chen added a comment -

          Doesn't usetimestamps="true" already do the trick?

          If you really don't want to depend on ant 1.8.0, you can use a condition-available trick as is already one with the ant-eclipse.jar.exists property elsewhere in the build file.

          Show
          Jolly Chen added a comment - Doesn't usetimestamps="true" already do the trick? If you really don't want to depend on ant 1.8.0, you can use a condition-available trick as is already one with the ant-eclipse.jar.exists property elsewhere in the build file.
          Hide
          Todd Lipcon added a comment -

          usetimestamps="true" doesn't appear to do it... perhaps it avoids re-downloading, but I think it still goes and at least sends a "HEAD" request.

          The <condition> is a good idea... perhaps I'll take a crack at it.

          Show
          Todd Lipcon added a comment - usetimestamps="true" doesn't appear to do it... perhaps it avoids re-downloading, but I think it still goes and at least sends a "HEAD" request. The <condition> is a good idea... perhaps I'll take a crack at it.
          Hide
          Plamen Jeliazkov added a comment -

          I went ahead and tried to do the <condition> for you. A little tricky due to the unless="offline" property. I think something like this is what you were looking for though. I may be a little off on my implementation.

          Show
          Plamen Jeliazkov added a comment - I went ahead and tried to do the <condition> for you. A little tricky due to the unless="offline" property. I think something like this is what you were looking for though. I may be a little off on my implementation.
          Hide
          Plamen Jeliazkov added a comment -

          Pick up a mistake. Should have been using an <isset> tag. Patch for review.

          Show
          Plamen Jeliazkov added a comment - Pick up a mistake. Should have been using an <isset> tag. Patch for review.
          Hide
          Konstantin Shvachko added a comment -

          Tried the patch offline. It still fails, but differently. It complains about resolving common jar.
          It would be good to have build working offline.

          Show
          Konstantin Shvachko added a comment - Tried the patch offline. It still fails, but differently. It complains about resolving common jar. It would be good to have build working offline.
          Hide
          Joep Rottinghuis added a comment -

          I solved this in a different way. Do a one-time build to prime the ~/.ivy2 directory and/or copy this from a machine with Internet access.

          I set the mvn.repo property in ~/build.properties (or pass it in as a -D option). Note that in common this property is called mvnrepo (no dot).
          mvn.repo=file:/home/jrottinghuis/buildrepo.
          Then I have two file:
          /home/jrottinghuis/buildrepo/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
          /home/jrottinghuis/buildrepo/org/apache/maven/maven-ant-tasks/2.0.10/maven-ant-tasks-2.0.10.jar

          The former is needed for all targets, the latter only if you want to use the mvn-install or mvn-publish targets.

          One other bootstrap problem with this is that the ivy and tasks cannot be found. I therefore manually copy both jars into hadoop-common/hdfs/ivy (also hadoop-common/common/ivy and hadoop-common/mapreduce). In Jenkins I have a simple build step for this.

          There is still a problem though, and that is that in the compile-contrib target a subant call is made. That does not pass along properties. That is problematic even when one sets other properties (for example hadoop-common.version. I'll file a separate bug for this.

          Show
          Joep Rottinghuis added a comment - I solved this in a different way. Do a one-time build to prime the ~/.ivy2 directory and/or copy this from a machine with Internet access. I set the mvn.repo property in ~/build.properties (or pass it in as a -D option). Note that in common this property is called mvnrepo (no dot). mvn.repo= file:/home/jrottinghuis/buildrepo . Then I have two file: /home/jrottinghuis/buildrepo/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar /home/jrottinghuis/buildrepo/org/apache/maven/maven-ant-tasks/2.0.10/maven-ant-tasks-2.0.10.jar The former is needed for all targets, the latter only if you want to use the mvn-install or mvn-publish targets. One other bootstrap problem with this is that the ivy and tasks cannot be found. I therefore manually copy both jars into hadoop-common/hdfs/ivy (also hadoop-common/common/ivy and hadoop-common/mapreduce). In Jenkins I have a simple build step for this. There is still a problem though, and that is that in the compile-contrib target a subant call is made. That does not pass along properties. That is problematic even when one sets other properties (for example hadoop-common.version. I'll file a separate bug for this.
          Hide
          Joep Rottinghuis added a comment -

          Forgot to mention that my I do pass
          resolvers=internal
          offline=true

          in the build.property as well.

          Show
          Joep Rottinghuis added a comment - Forgot to mention that my I do pass resolvers=internal offline=true in the build.property as well.
          Hide
          Joep Rottinghuis added a comment -

          Three files are needed locally to build w/o Inernet connection.

          Set the following property
          mvn.repo=/home/user

          and have three files available:
          /home/user/buildrepo/dist/commons/daemon/binaries/1.0.2/linux/commons-daemon-1.0.2-bin-linux-i386.tar.gz
          /home/user/buildrepo/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
          /home/user/buildrepo/org/apache/maven/maven-ant-tasks/2.0.10/maven-ant-tasks-2.0.10.jar

          Show
          Joep Rottinghuis added a comment - Three files are needed locally to build w/o Inernet connection. Set the following property mvn.repo=/home/user and have three files available: /home/user/buildrepo/dist/commons/daemon/binaries/1.0.2/linux/commons-daemon-1.0.2-bin-linux-i386.tar.gz /home/user/buildrepo/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar /home/user/buildrepo/org/apache/maven/maven-ant-tasks/2.0.10/maven-ant-tasks-2.0.10.jar
          Hide
          Joep Rottinghuis added a comment -

          With the properties I described and the patch in HDFS-2211 (and without the patched attached to this Jira) I can successfully build hdfs on a machine w/o Internet connectivity.

          Show
          Joep Rottinghuis added a comment - With the properties I described and the patch in HDFS-2211 (and without the patched attached to this Jira) I can successfully build hdfs on a machine w/o Internet connectivity.

            People

            • Assignee:
              Joep Rottinghuis
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development