Hive
  1. Hive
  2. HIVE-1120

In ivy offline mode, don't delete downloaded jars

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.6.0
    • Component/s: Build Infrastructure
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Once jars have been successfully downloaded, it would be good for developer productivity to be able to do

      ant -Doffline=true clean && package

      Currently this won't work because the clean will delete the downloaded jars.

      1. HIVE-1120.2.patch
        5 kB
        John Sichi
      2. HIVE-1120.1.patch
        4 kB
        John Sichi

        Issue Links

          Activity

          Hide
          Zheng Shao added a comment -

          Committed. Thanks John!

          Show
          Zheng Shao added a comment - Committed. Thanks John!
          Hide
          John Sichi added a comment -

          Resubmitting since this is the expected behavior.

          Show
          John Sichi added a comment - Resubmitting since this is the expected behavior.
          Hide
          John Sichi added a comment -

          The behavior you're seeing is expected. In step 4, it is not actually downloading anything from the network; it is copying/unpacking the archives from ~/.ant/cache to hive/build.
          This is necessary since we delete everything exception build/ivy as part of ant clean.

          You can verify this by disconnecting your network cable before step 4; it will succeed (with a bunch of warning complaints for when it tries to access the remote repo for version checks).

          Show
          John Sichi added a comment - The behavior you're seeing is expected. In step 4, it is not actually downloading anything from the network; it is copying/unpacking the archives from ~/.ant/cache to hive/build. This is necessary since we delete everything exception build/ivy as part of ant clean. You can verify this by disconnecting your network cable before step 4; it will succeed (with a bunch of warning complaints for when it tries to access the remote repo for version checks).
          Hide
          Zheng Shao added a comment -

          I did the following

          1. ant clean
          2. ant package
          3. ant -Doffline=true clean
          4. ant -Doffline=true package
          5. ant -Doffline=true package
          

          "ant" is checking/downloading hadoop source in command 4. However command 5 is fine: we don't download hadoop source again.
          John, can you post the sequence of commands you run to make this work?

          Show
          Zheng Shao added a comment - I did the following 1. ant clean 2. ant package 3. ant -Doffline= true clean 4. ant -Doffline= true package 5. ant -Doffline= true package "ant" is checking/downloading hadoop source in command 4. However command 5 is fine: we don't download hadoop source again. John, can you post the sequence of commands you run to make this work?
          Hide
          Zheng Shao added a comment -

          Will test and commit.

          Show
          Zheng Shao added a comment - Will test and commit.
          Hide
          Carl Steinbach added a comment -

          +1

          Be forewarned that if you have manually installed an older version of
          Ivy in $ANT_HOME/lib this copy will get picked up in preference to downloading
          the newer version. I saw the following error until I deleted the older Ivy JAR
          in my $ANT_HOME/lib directory:

          ivy-retrieve-hadoop-source:
          [ivy:retrieve] :: Ivy 2.0.0-rc2 - 20081028224207 :: http://ant.apache.org/ivy/ ::
          :: loading settings :: file = /Users/carl/Projects/hd9/hive-trunk/ivy/ivysettings.xml
          
          BUILD FAILED
          /Users/carl/Projects/hd9/hive-trunk/build.xml:148: The following error occurred while executing this line:
          /Users/carl/Projects/hd9/hive-trunk/build.xml:93: The following error occurred while executing this line:
          /Users/carl/Projects/hd9/hive-trunk/shims/build.xml:55: The following error occurred while executing this line:
          /Users/carl/Projects/hd9/hive-trunk/build-common.xml:180: impossible to configure ivy:settings with given file: /Users/carl/Projects/hd9/hive-trunk/ivy/ivysettings.xml : java.text.ParseException: failed to load settings from file:/Users/carl/Projects/hd9/hive-trunk/ivy/ivysettings.xml: impossible to set defaultTTL to eternal on class org.apache.ivy.core.cache.DefaultRepositoryCacheManager
          
          Show
          Carl Steinbach added a comment - +1 Be forewarned that if you have manually installed an older version of Ivy in $ANT_HOME/lib this copy will get picked up in preference to downloading the newer version. I saw the following error until I deleted the older Ivy JAR in my $ANT_HOME/lib directory: ivy-retrieve-hadoop-source: [ivy:retrieve] :: Ivy 2.0.0-rc2 - 20081028224207 :: http://ant.apache.org/ivy/ :: :: loading settings :: file = /Users/carl/Projects/hd9/hive-trunk/ivy/ivysettings.xml BUILD FAILED /Users/carl/Projects/hd9/hive-trunk/build.xml:148: The following error occurred while executing this line: /Users/carl/Projects/hd9/hive-trunk/build.xml:93: The following error occurred while executing this line: /Users/carl/Projects/hd9/hive-trunk/shims/build.xml:55: The following error occurred while executing this line: /Users/carl/Projects/hd9/hive-trunk/build-common.xml:180: impossible to configure ivy:settings with given file: /Users/carl/Projects/hd9/hive-trunk/ivy/ivysettings.xml : java.text.ParseException: failed to load settings from file:/Users/carl/Projects/hd9/hive-trunk/ivy/ivysettings.xml: impossible to set defaultTTL to eternal on class org.apache.ivy.core.cache.DefaultRepositoryCacheManager
          Hide
          John Sichi added a comment -

          .2.patch resolves conflicts with checkstyle change.

          Show
          John Sichi added a comment - .2.patch resolves conflicts with checkstyle change.
          Hide
          Zheng Shao added a comment -

          Hi John, it seems the recent commits of "ant checkstyle" make this patch conflicting with trunk.
          Can you regenerate the patch?

          Show
          Zheng Shao added a comment - Hi John, it seems the recent commits of "ant checkstyle" make this patch conflicting with trunk. Can you regenerate the patch?
          Hide
          John Sichi added a comment -

          Once you have this patch and have run ivy to download all dependencies, you can set ANT_ARGS="-Doffline=true" in your environment for normal use, then unset it only when you need to let ivy download updated dependencies.

          Show
          John Sichi added a comment - Once you have this patch and have run ivy to download all dependencies, you can set ANT_ARGS="-Doffline=true" in your environment for normal use, then unset it only when you need to let ivy download updated dependencies.
          Hide
          John Sichi added a comment -

          Here's the ivy upgrade patch.

          After applying this patch, you need to do

          cd ~
          rm -rf .ant

          Otherwise you'll get a build error from your old cached ivy version. This is necessary regardless of whether you're using offline mode or not. Once this patch is committed, I'll broadcast a notice to that effect to hive-dev.

          Besides the ivy upgrade, here are some other notes on the change:

          • In online mode (the default), everything should continue to work exactly as before (ant clean will blow away the entire build directory)
          • In offline mode, ant clean will leave behind build/ivy (deleting everything else under build) to avoid the need to re-download ivy on the next sync
          • In offline mode, on the first "ant package" after "ant clean", ant will re-expand archives from the ~/.ant/cache into the build dir, thus avoiding a download, although it will still exchange a few messages with remote repos--but if the remote repos are not available, it will keep going with warnings, so once you've downloaded everything, you can go off the grid to hack away disconnected while still using ant clean as needed
          • In offline mode, repeated execution of "ant package" (without "ant clean" in the middle) will skip ivy entirely, same as before this patch
          Show
          John Sichi added a comment - Here's the ivy upgrade patch. After applying this patch, you need to do cd ~ rm -rf .ant Otherwise you'll get a build error from your old cached ivy version. This is necessary regardless of whether you're using offline mode or not. Once this patch is committed, I'll broadcast a notice to that effect to hive-dev. Besides the ivy upgrade, here are some other notes on the change: In online mode (the default), everything should continue to work exactly as before (ant clean will blow away the entire build directory) In offline mode, ant clean will leave behind build/ivy (deleting everything else under build) to avoid the need to re-download ivy on the next sync In offline mode, on the first "ant package" after "ant clean", ant will re-expand archives from the ~/.ant/cache into the build dir, thus avoiding a download, although it will still exchange a few messages with remote repos--but if the remote repos are not available, it will keep going with warnings, so once you've downloaded everything, you can go off the grid to hack away disconnected while still using ant clean as needed In offline mode, repeated execution of "ant package" (without "ant clean" in the middle) will skip ivy entirely, same as before this patch
          Hide
          John Sichi added a comment -

          Yes, 2.1.0 was released in Oct 2009. I'll look into the upgrade.

          Show
          John Sichi added a comment - Yes, 2.1.0 was released in Oct 2009. I'll look into the upgrade.
          Hide
          Zheng Shao added a comment -

          Is ivy 2.1.0 released? I guess it's not a big problem to upgrade ivy.

          Show
          Zheng Shao added a comment - Is ivy 2.1.0 released? I guess it's not a big problem to upgrade ivy.
          Hide
          John Sichi added a comment -

          I looked into doing this via ant changes, but it would involve selectively preserving artifact copies under hive's build directory via excludes in our clean target.

          A much better solution is available in IVY-879, but it involves upgrading ivy to 2.1.0. It's better because it allows us to continue blowing away the build directories entirely and then just refresh from the local cache, without hitting the remote repository.

          Show
          John Sichi added a comment - I looked into doing this via ant changes, but it would involve selectively preserving artifact copies under hive's build directory via excludes in our clean target. A much better solution is available in IVY-879 , but it involves upgrading ivy to 2.1.0. It's better because it allows us to continue blowing away the build directories entirely and then just refresh from the local cache, without hitting the remote repository.

            People

            • Assignee:
              John Sichi
              Reporter:
              John Sichi
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development