Ivy
  1. Ivy
  2. IVY-1159

Resolved Ivy Properties written to cache during ivy:resolve incorrectly represents transitive evictions

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0-RC1
    • Fix Version/s: 2.2.0-RC1
    • Component/s: Core
    • Labels:
      None

      Description

      In ResolveEngine.resolve the code that writes the properties file appears to be incorrect.
      When the dependencies collection includes two or more entries for the same dependency (one from the root ivy file and the others being transitives), the properties file will always only ever be populated with the information from the IvyNode that belongs to the root ivy file (effectively the comment states this, so it certainly appears intentional).

      This produces the following bugs / undesirable effects:

      • incorrect delivered descriptor in some situations (where a dependency is both directly and transitively imported)
      • order of declaration of dependencies alters behaviour
        These are pretty major problems because (as demonstrated below) a delivered/published ivy descriptor can identify completely bogus revisions for the dynamic->static replacements, which are not to the actual revisions used when compiling etc

      Set up
      module one has no dependencies and has been published (as rev = 1, say)
      module two has a dependency on module one (only) and has been published (as rev = 1, also)
      module three has dependencies on both modules one and two and lists them in the order two,one
      In the module three descriptor the revisions mentioned for two and one do not actually exist (two=0 & one=0 say - repository only contains rev=1 of both)
      Ivy settings (attached) has a resolver that refers to a local repo, and has force=true and local=true set.

      Problem Case - use case: publishing module 3
      1) ivy:resolve on module three, using refresh=true, transitive=true. Otherwise nothing special here.
      then
      2) ivy:deliver (status=reelase, pubdate=now, deliverpattern supplied, pubrevision supplied (rev=1))
      3) (lastly, the ivy:publish step woud happen here, but is not relevant to the problem)

      Expected outcome:
      Because the primary resolver has force=true, rev 1 of both module dependencies of 'three' should be selected [Because rev=1 publications are the only ones present in the repository] when resolving three's declared deps (both of which declare a dep on rev=0).

      Actual outcome:
      Delivered descriptor correctly names two as being resolved to rev=1. Incorrectly names module one as being resolved to rev=0 (which doesn't exist - and never has!)

      Workaround:
      Reverse the order of the declared dependencies in three's ivy descriptor (i.e new order is one, two) - problem does not occur.

      Diagnosis:
      Logs for resolve appear to show everything is fine (the eviction is noted, the generated report xml shows the declared direct dependency (one, rev=0) being evicted by the transitive dependency (one, rev=1 as declared in two's published ivy descriptor)
      However, debugging DeliverEngine.java and ResolveEngine.java - it is apparent that the 'resolved ivy properties file' is used to drive the replacement of dynamic revisions with static ones during deliver. In the error case, the properties file shows this:
      +revision\:#@#\:+0\:#@#\:+module\:#@#\:+one\: <snip> :=0 ?
      +revision\:#@#\:+0\:#@#\:+module\:#@#\:+two\: <snip> :=1 release
      The '?' is put there because the IvyNode that represented the direct dependency has no descriptor (correctly, because it has been evicted). Because the real selected revision is not put into the properties file, the ivy:deliver task is doomed to produce wrong results.

      The correct behaviour in this case would be for the EvictionInfo stored against the evicted IvyNode to be inspected to find the appropriate revision information. I have attached a pretty ugly example of how this could be achieved.

      Just before the comment ("The evicted modules have no description, so we can't put their status")

                                  if (depDescriptor == null) {
                                    EvictionData ed = null;
                                    for (int j=0; j<confs.length && ed == null; j++) {
                                      ed = dependencies[i].getEvictedData(confs[j]);
                                    }
                                    if (ed != null) {
                                      Collection selected = ed.getSelected();
                                      if (selected != null) {
                                        if (selected.size() == 1) {
                                          IvyNode sel = (IvyNode)selected.iterator().next();
                                          depDescriptor = sel.getDescriptor();
                                          depResolvedId = sel.getResolvedId();
                                          if (depResolvedId == null) {
                                            throw new NullPointerException("getResolvedId() is null for (transitive) " 
                                              + dependencies[i].toString());
                                          }
                                          rev = depResolvedId.getRevision();
                                        }
                                        else if (selected.size() > 1) {
                                          throw new RuntimeException("Unexpectedly large number of selecteds within eviction collection");
                                        }
                                      }
                                    }
                                  }
      

        Issue Links

          Activity

          Hide
          Maarten Coene added a comment -

          I think such a change would be too big to make it into the next release.

          So I've fixed it like this:

          • the "replaceDynamicRev=true" setting will replace dynamic revisions (not static ones) with the resolved revision
          • a new "replaceForcedRev=true" setting will update static/dynamic revisions with the forced revision (defaults to 'false' for backward compatibility)

          the replacement of static evicted revisions as I suggested earlier isn't possible because this depends on the configurations being resolved.

          So in your case, you should call ivy:deliver like this:

          <ivy:deliver replaceDynamicRev="true" replaceForcedRev="true" ... />
          

          Hope this solves your problem.
          Maarten

          Show
          Maarten Coene added a comment - I think such a change would be too big to make it into the next release. So I've fixed it like this: the "replaceDynamicRev=true" setting will replace dynamic revisions (not static ones) with the resolved revision a new "replaceForcedRev=true" setting will update static/dynamic revisions with the forced revision (defaults to 'false' for backward compatibility) the replacement of static evicted revisions as I suggested earlier isn't possible because this depends on the configurations being resolved. So in your case, you should call ivy:deliver like this: <ivy:deliver replaceDynamicRev= " true " replaceForcedRev= " true " ... /> Hope this solves your problem. Maarten
          Hide
          Ed Burcher added a comment - - edited

          Hi Maarten - you raise some interesting points. The question of whether a revision is treated as being static or dynamic would therefore appear to be a combination of the Revision Spec (1.0 vs 1.0+ etc), and the resolver properties. Secondly, let's assert that the delivery engine needs to understand whether something was treated as static or dynamic to effect the correct delivery behaviour (dynamic being something that should be replaced with the resolved version).
          Because of this, I'd argue that the static/dynamic determination that was 'current' whilst obtaining each selected revision should be retained as part of the retained resolve state, so that it can be used by the delivery engine. In other words, the resolver that was used to obtain the ultimately selected revision should contribute state to the delivery engine specifically because the static/dynamic treatment can only be fully understood with reference to a particular resolver.


          Your second bullet is again very interesting and I have to say that personally I would make the same argument for static revisions:
          Presumably the situation described will only occur with static revisions where transitivity is enabled during resolve. Is it possible that the original vision of a distinction between static and dynamic revisions was made substantially less clear by the introduction of transitive resolution in 1.4+ ?
          I'd argue this because it's clear that when transitive resolution is not used, behaviour is basically consistent. Static revs are not replaced in deliver, whereas dynamic ones are. As soon as you go with transitive resolution, evictions become possible both for 'static' and 'dynamic' revisions. (Moreover different ivy descriptors in the overall set may have different revision specs - some of which may be static and some dynamic). Whereas pre-1.4 ivy offers a delivered descriptor that is consistent with the resolve (for any combination of static/dynamic), your bullet illustrates that this is not the case anymore. Personally I am not convinced the introduction of transitive resolution should have had any bearing on whether deliver is supposed to in principle produce a file that allows an identical, static, repeatable resolution in future.

          In practice I believe that using transitive resolution makes any and all revisions evictable and hence dynamic. I therefore propose the following pseudo logic:

          IF resolve mode is TRANSITIVE
          treat all dependencies as dynamic when emitting delivered ivy file
          ELSE
          for each dependency:
          IF the specification is dynamic OR if the resolver that selected the revision is forced-dynamic, then treat as dynamic in the delivery stage
          ELSE treat as static


          Regarding your suggestion - I think a new flag could be a pragmatic way to solve this in the short term. However I am not convinced it will work as stated - the properties file that drives the replacement in ivy:deliver is actually written in the ivy:resolve stage (when the bug occurs, this file contains bogus data). Since the resolve task does not have knowledge of the flags that will be passed to the deliver task, the written data would still be bad. This is basically because it treats a rev as static when it should have been treated dynamically given that it existed within a transitive, forced-resolve context.

          Show
          Ed Burcher added a comment - - edited Hi Maarten - you raise some interesting points. The question of whether a revision is treated as being static or dynamic would therefore appear to be a combination of the Revision Spec (1.0 vs 1.0+ etc), and the resolver properties. Secondly, let's assert that the delivery engine needs to understand whether something was treated as static or dynamic to effect the correct delivery behaviour (dynamic being something that should be replaced with the resolved version). Because of this, I'd argue that the static/dynamic determination that was 'current' whilst obtaining each selected revision should be retained as part of the retained resolve state, so that it can be used by the delivery engine. In other words, the resolver that was used to obtain the ultimately selected revision should contribute state to the delivery engine specifically because the static/dynamic treatment can only be fully understood with reference to a particular resolver. – Your second bullet is again very interesting and I have to say that personally I would make the same argument for static revisions: Presumably the situation described will only occur with static revisions where transitivity is enabled during resolve. Is it possible that the original vision of a distinction between static and dynamic revisions was made substantially less clear by the introduction of transitive resolution in 1.4+ ? I'd argue this because it's clear that when transitive resolution is not used, behaviour is basically consistent. Static revs are not replaced in deliver, whereas dynamic ones are. As soon as you go with transitive resolution, evictions become possible both for 'static' and 'dynamic' revisions. (Moreover different ivy descriptors in the overall set may have different revision specs - some of which may be static and some dynamic). Whereas pre-1.4 ivy offers a delivered descriptor that is consistent with the resolve (for any combination of static/dynamic), your bullet illustrates that this is not the case anymore. Personally I am not convinced the introduction of transitive resolution should have had any bearing on whether deliver is supposed to in principle produce a file that allows an identical, static, repeatable resolution in future. In practice I believe that using transitive resolution makes any and all revisions evictable and hence dynamic. I therefore propose the following pseudo logic: IF resolve mode is TRANSITIVE treat all dependencies as dynamic when emitting delivered ivy file ELSE for each dependency: IF the specification is dynamic OR if the resolver that selected the revision is forced-dynamic, then treat as dynamic in the delivery stage ELSE treat as static – Regarding your suggestion - I think a new flag could be a pragmatic way to solve this in the short term. However I am not convinced it will work as stated - the properties file that drives the replacement in ivy:deliver is actually written in the ivy:resolve stage (when the bug occurs, this file contains bogus data). Since the resolve task does not have knowledge of the flags that will be passed to the deliver task, the written data would still be bad. This is basically because it treats a rev as static when it should have been treated dynamically given that it existed within a transitive, forced-resolve context.
          Hide
          Maarten Coene added a comment -

          Ed, I understand your concerns.

          • Regarding the force/dynamic style behaviour: this is only true within the context of the resolver in force mode. If this resolver doesn't find the module for instance, the next resolver is used but the revision is no longer 'latest.integration'. The same argument can be made when these properties are written out: this happens after the resolve process, no longer within the scope of the resolver in forced mode, which means Ivy will consider the dependency having a static revision again.
          • Regarding the deliverd ivy.xml representing the resolution results. You can make the same argument for evicted revisions. Static evicted revisions in the delivered ivy.xml will also not get replaced by the resolved revisions. Maybe we could add an extra attribute to the deliver Ant task indicating that static revisions should get replaced as well? Something like:
          <ivy:deliver replaceDynamicRev="true" replaceStaticRev="true" />
          

          The default value of this 'replaceStaticRev' attribute would be false to keep backwards compatibility. What do you think?

          Show
          Maarten Coene added a comment - Ed, I understand your concerns. Regarding the force/dynamic style behaviour: this is only true within the context of the resolver in force mode. If this resolver doesn't find the module for instance, the next resolver is used but the revision is no longer 'latest.integration'. The same argument can be made when these properties are written out: this happens after the resolve process, no longer within the scope of the resolver in forced mode, which means Ivy will consider the dependency having a static revision again. Regarding the deliverd ivy.xml representing the resolution results. You can make the same argument for evicted revisions. Static evicted revisions in the delivered ivy.xml will also not get replaced by the resolved revisions. Maybe we could add an extra attribute to the deliver Ant task indicating that static revisions should get replaced as well? Something like: <ivy:deliver replaceDynamicRev="true" replaceStaticRev="true" /> The default value of this 'replaceStaticRev' attribute would be false to keep backwards compatibility. What do you think?
          Hide
          Ed Burcher added a comment -

          Thanks for taking a look. However your assessment puzzles me. My understanding of a resolver having force=true is that every revision in the ivy file is treated as dynamic. That's certainly my reading of this extract from http://ant.apache.org/ivy/history/trunk/settings/resolvers.html#common
          "
          (under heading) Force

          Any standard resolver can be used in force mode, which is used mainly to handle local development builds. In force mode, the resolver attempts to find a dependency whatever the requested revision is (internally it replace the requested revision by 'latest.integration'), and if it finds one, it forces this revision to be returned, even when used in a chain with returnFirst=false.
          "
          As such I'd expect to see dynamic-style behaviour on deliver in this case.

          I'd also make the same argument by suggesting that it 'always makes sense' that the deliver task writes a published ivy file that represents the resolution that just took place. After all, the point about dynamic and static ivy files is that the dynamic ones become static at the point where you want to fix the revision identities (Because you want repeatable, unchanging, predictable resolution behaviour for published artifacts). It would seem strange to me for the resolve step to force resolution to some acceptable artifacts, but for ivy not to allow a repeatable, static, resolve of the same in future by not supporting generation of a delivered ivy file to represent what was resolved. Even weirder would be if the deliver task produced an ivy file containing static revisions that may not even exist (as in my test case) and certainly were not the ones actually resolved.

          Show
          Ed Burcher added a comment - Thanks for taking a look. However your assessment puzzles me. My understanding of a resolver having force=true is that every revision in the ivy file is treated as dynamic. That's certainly my reading of this extract from http://ant.apache.org/ivy/history/trunk/settings/resolvers.html#common " (under heading) Force Any standard resolver can be used in force mode, which is used mainly to handle local development builds. In force mode, the resolver attempts to find a dependency whatever the requested revision is (internally it replace the requested revision by 'latest.integration'), and if it finds one, it forces this revision to be returned, even when used in a chain with returnFirst=false. " As such I'd expect to see dynamic-style behaviour on deliver in this case. I'd also make the same argument by suggesting that it 'always makes sense' that the deliver task writes a published ivy file that represents the resolution that just took place. After all, the point about dynamic and static ivy files is that the dynamic ones become static at the point where you want to fix the revision identities (Because you want repeatable, unchanging, predictable resolution behaviour for published artifacts). It would seem strange to me for the resolve step to force resolution to some acceptable artifacts, but for ivy not to allow a repeatable, static, resolve of the same in future by not supporting generation of a delivered ivy file to represent what was resolved. Even weirder would be if the deliver task produced an ivy file containing static revisions that may not even exist (as in my test case) and certainly were not the ones actually resolved.
          Hide
          Maarten Coene added a comment -

          I don't think that the delivered ivy file should contain rev=1, but it should still contain rev=0 for both your dependencies.
          The reason is that only dynamic revisions should get replaced, not static ones (even if the are evicted by another one).

          I'm working on a fix so that in the case of a forced revision it doesn't get replaced.

          Show
          Maarten Coene added a comment - I don't think that the delivered ivy file should contain rev=1, but it should still contain rev=0 for both your dependencies. The reason is that only dynamic revisions should get replaced, not static ones (even if the are evicted by another one). I'm working on a fix so that in the case of a forced revision it doesn't get replaced.
          Hide
          Ed Burcher added a comment -

          still a live bug as of SVN 940638 4th May 2010

          uploaded test case to aid matters

          Show
          Ed Burcher added a comment - still a live bug as of SVN 940638 4th May 2010 uploaded test case to aid matters
          Hide
          Ed Burcher added a comment -

          It turns out that things are also sensitive to whether the (non-existent) revision in the root ivy file is lower or higher numerically than the real one. This may be because of ordering of the IvyNodes in the dependencies collection. I hadn't noticed this when I was simplifying my test case.

          => Replaced refs to rev=2 in the above text to say rev=0.

          Show
          Ed Burcher added a comment - It turns out that things are also sensitive to whether the (non-existent) revision in the root ivy file is lower or higher numerically than the real one. This may be because of ordering of the IvyNodes in the dependencies collection. I hadn't noticed this when I was simplifying my test case. => Replaced refs to rev=2 in the above text to say rev=0.

            People

            • Assignee:
              Maarten Coene
              Reporter:
              Ed Burcher
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development