Ivy
  1. Ivy
  2. IVY-1252

symlinkmass feature based on symlink feature of ivy:retrieve

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.4.0-RC1
    • Component/s: Ant, Core, Documentation
    • Labels:
    • Environment:

      UNIX/Linux

      Description

      I have an improvement I have made in a locally built copy of Ivy, to speed up the making of symbolic links when retrieving dependencies. It was written based on:
      apache-ivy-2.2.0-src.tar.gz
      It changes existing files, but creates no new files. I have already made and am using the change in our local build system, and want to contribute back so that others can use it.

      ....

      The documentation for this feature would be added to:
      http://ant.apache.org/ivy/history/latest-milestone/use/retrieve.html
      in the main table:
      Attribute
      symlinkmass
      Description
      true to create symbolic links in mass, false to copy the artifacts.
      symlinkmass overrides "symlink" if both are set to "true".
      symlinkmass will create the same symbolic links "symlink" does,
      but with a single process call to "sh" with batched "ln" commands passed in as standard input.
      (works on UNIX/Linux, on other systems you may need to script it)
      Far large lists of resolved jars, this can be dramatically faster.
      The destination of the symbolic links depends on the value of the useOrigin attribute
      The events "StartRetrieveArtifactEvent" and EndRetrieveEvent,
      are NOT fired by this activity, because it is not clear when they should be called.
      Required
      No. Defaults to false

      ....

      Non trivial code changes are made in:
      apache-ivy-2.2.0/src/java/org/apache/ivy/core/retrieve/RetrieveEngine.java
      branching in the code based on the "symlinkmass" boolean
      in existing retrieval loop (where fetching of files would normally occur) to collect destination and source paths
      and after the loop to do the mass fetch
      import added for java.util.HashMap

      A non trivial method is added to handle linking of files based on a Map of destinations and sources. Existing methods are untouched.
      apache-ivy-2.2.0/src/java/org/apache/ivy/util/FileUtil.java
      a method to do the symlinking based of a Map of the destinations to sources
      imports added for java.util.Iterator, java.util.LinkedList, java.util.Map, java.util.regex.Pattern

      Trivial additions are made to pass the control parameter "symlinkmass"
      apache-ivy-2.2.0/src/java/org/apache/ivy/core/retrieve/RetrieveOptions.java
      apache-ivy-2.2.0/src/java/org/apache/ivy/ant/IvyRetrieve.java
      apache-ivy-2.2.0/src/java/org/apache/ivy/Main.java

      A unit test is added in
      apache-ivy-2.2.0/test/java/org/apache/ivy/core/retrieve/RetrieveTest.java
      basically a duplicate of the unit test for "symlink" with a few internal code comments

      ...

      Notes on why we want this:

      The build system I maintain (Ant + Ivy, with Linux scripts for Developer support) was using the "symlink" feature for ivy:retrieve, because over a complete build of our code modules (about 25 separate and growing) the sub-builds make a total of about 3000 dependency links and the file space savings made it complete worth it. The build system is not about speed. It is about modularity, and we use Ivy to link the modules in a dependency tree so that we can swap them out as we . We like Ivy very much for that. Thank you all.

      But, while speed is a secondary consideration, it is not unimportant... and this improvement cuts about 8 minutes off an 18 minute build.

        Activity

        Gene Smith created issue -
        Hide
        Gene Smith added a comment -

        The source for the feature. And text which would go into the documentation with it.

        Show
        Gene Smith added a comment - The source for the feature. And text which would go into the documentation with it.
        Gene Smith made changes -
        Field Original Value New Value
        Attachment apache-ivy-2.2.0-src-for-symlinkmass.tar [ 12465159 ]
        Attachment ivy-docs-update-text.txt [ 12465160 ]
        Nicolas Lalevée made changes -
        Labels patch
        Hide
        Nicolas Lalevée added a comment -

        "patch" applied. I have a little rework it so the code is simpler, but the functionality is the same.

        Show
        Nicolas Lalevée added a comment - "patch" applied. I have a little rework it so the code is simpler, but the functionality is the same.
        Nicolas Lalevée made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Assignee Nicolas Lalevée [ hibou ]
        Fix Version/s trunk [ 12320744 ]
        Resolution Fixed [ 1 ]
        Maarten Coene made changes -
        Fix Version/s 2.4.0-RC1 [ 12326648 ]
        Fix Version/s trunk [ 12320744 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        621d 21h 28m 1 Nicolas Lalevée 15/Aug/12 18:34

          People

          • Assignee:
            Nicolas Lalevée
            Reporter:
            Gene Smith
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 2h
              2h
              Remaining:
              Remaining Estimate - 2h
              2h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development