IvyDE
  1. IvyDE
  2. IVYDE-43

[PATCH] New resolver allows project linking within the workspace

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: classpath container
    • Labels:
      None

      Description

      I've been dreaming and tinkering with this idea for a while, and it finally evolved into these changes.
      This introduces a new plugin - called WorkspaceResolver.

      DISCLAIMER: These changes should NOT change how IVYDE works unless the new resolver is set up in the Ivy configuration file. Therefore, it should be safe to use with any existing Ivy configuration. The only potential issue would occur if you are using artifacts of type "project", since I'm treating those differently.

      When used with my custom IvyClasspathContainer changes, the WorkspaceResolver will link dependent projects when they are open in the same workspace, allowing full-fledged linked project functionality Eclipse provides, such as incremental compilation, debugging, mouseover javadoc pop-ups, and source browsing across projects.

      The resolver will not work until it is properly configured, see instructions below.

      I attached the patch against IvyDE 1.2 rev. 605. It also includes the minor console logging changes I submitted with IVYDE-39. Please let me know what you think and if there are any major issues or suggestions.

      How it works:

      During a resolve, if the WorkspaceResolver is in the resolve chain (see below how to set it up), it looks at all open projects in the workspace that have Ivy containers.
      The first project that publishes the module on which the project being resolved depends, will be picked and returned as a special type of artifact called "project".
      The IvyClasspathContainer will recognize the artifact as a project and put the eclipse project as a dependent project within the classpath container of the parent. I will attach the screenshot to show this.

      If you do not want a project to be linked as a dependency, close it or delete from the workspace.
      As soon as you do that, any projects that were linked to it will automatically re-resolve (using WorkspaceResourceChangeListener) (if that preference is enabled) and use the standard Ivy means of finding the dependency.

      I am also planning to have the WorkspaceResourceChangeListener auto-resolve when a new project is added or opened, so opening a project will automatically link it into the currently open projects where necessary. This has been problematic, since I cannot seem to catch the right event when a project is open.

      Some limitations and potential issues:

      1) Since the resolver is not aware which module revision an eclipse project is publishing (we never put the version inside the info tag of a project's ivy.xml), it optimistically matches any revision of the module.

      2) Since the resolver stops after finding the first open project which matches the module, having multiple open versions of the same project in the workspace (for example, different branches) may set the wrong version as a dependency.
      You are advised to only open the version of the project which you want other projects in the workspace to depend on.

      3) NOTE: Transitive dependencies are not passed from the dependent project to the parent when projects are linked. If you find you are missing some transitive dependencies, just set your dependent eclipse project to export its ivy dependencies.
      (Project->Properties->Java Build Path->Order and Export-> check the ivy container)
      This will only export the configuration(s) that project is using and not what a dependent project may ask for when it's being resolved. To do that, this resolver will need to be modified to pass transitive dependencies along.

      How to set it up

      Set up a custom resolver in the resolve chain (see example below),

      Usage Example:

      Suppose you have a main ivyconf.xml which sets up all your resolvers to be
      executed at command-line.

      1) Create a new file which includes your main ivyconf.xml file
      2) Define the custom eclipse resolver type.
      3) Create a simple chain which uses the eclipse resolver first and then
      the resolver which would normally be used in your config file.
      It could be the default resolver or any other, depending on your situation.
      4) (Optional) To optimize, if you know you will only be developing modules
      in your organization, you may limit the calls to the eclipse resolver
      to only modules in your organization, by using the module directives.

      IMPORTANT NOTE: If the main config file already restricts modules in your organization
      to a specific resolver, you will not be able to override it in your own file.

      Either completely duplicate the main file and add eclipse resolver there,
      or have a base file without any org-specific restrictions and then two
      files which include it: the eclipse config and the main config.

      (See Ivy manual for your reference).
      5) In Eclipse preferences, navigate to Ivy preferences and select the
      new config file just created as your default configuration file.

      Sample ivyconf-eclipse.xml:

      <ivyconf>

      <include file="/path/to/ivyconf.xml"/>

      <typedef name="eclipse" classname="org.jayasoft.ivyde.eclipse.resolver.WorkspaceResolver"/>

      <resolvers>
      <eclipse name="eclipse"/>
      <chain name="eclipse-chain" returnFirst="true">
      <resolver ref="eclipse"/>
      <resolver ref="originalResolver"/>
      </chain>
      </resolvers>

      <modules>
      <module organisation="myOrg" name="." resolver="eclipse-chain"/>
      </modules>

      </ivyconf>

      1. ASF.LICENSE.NOT.GRANTED--ivyde-43.patch
        73 kB
        Eugene Goldfarb
      2. ASF.LICENSE.NOT.GRANTED--ivyde-example-censored.JPG
        74 kB
        Eugene Goldfarb
      3. ASF.LICENSE.NOT.GRANTED--ivyde-example-censored.JPG
        74 kB
        Eugene Goldfarb
      4. IVYDE-43-r641334.patch
        37 kB
        Nicolas Lalevée

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          612d 3h 42m 1 Nicolas Lalevée 03/Aug/08 15:42
          Nicolas Lalevée made changes -
          Link This issue duplicates IVYDE-89 [ IVYDE-89 ]
          Nicolas Lalevée made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Assignee Xavier Hanin [ xavier ]
          Resolution Duplicate [ 3 ]
          Hide
          Nicolas Lalevée added a comment - - edited

          Finally the solution from IVYDE-89 has been integrated.

          Show
          Nicolas Lalevée added a comment - - edited Finally the solution from IVYDE-89 has been integrated.
          Hide
          Matt Goldspink added a comment -

          Hi Nicolas,

          Thanks for pointing out the other JIRA. It was not as good as the patch attached to this issue but I do see your reasoning for not wanting to enforce a different ivysettings.xml file for working in Eclipse. So I went ahead and merged pieces from the two patches and also added configuration options to turn on or off workspace resolution on a per project basis. I've uploaded this patch to [https://issues.apache.org/jira/browse/IVYDE-89] so others can try it out. Our team has a lot of interdependent projects and we've all been using the patch for a few months and it has been working great. We would love to see this feature added to IvyDE.

          Show
          Matt Goldspink added a comment - Hi Nicolas, Thanks for pointing out the other JIRA. It was not as good as the patch attached to this issue but I do see your reasoning for not wanting to enforce a different ivysettings.xml file for working in Eclipse. So I went ahead and merged pieces from the two patches and also added configuration options to turn on or off workspace resolution on a per project basis. I've uploaded this patch to [ https://issues.apache.org/jira/browse/IVYDE-89 ] so others can try it out. Our team has a lot of interdependent projects and we've all been using the patch for a few months and it has been working great. We would love to see this feature added to IvyDE.
          Hide
          Nicolas Lalevée added a comment -

          To properly fix the classpath issues, a simple fix will be needed in Ivy: allow to set the classloader used by Ivy to instanciate the class specified in the ivysettings.xml.
          But even if we impelemnt that fix, we are not sure to include the WorkspaceResolver in IvyDE as it force you to modify your settings. See previous comments.

          But you might be interested in IVYDE-89, which suggest a simpler solution.
          I did not tested it thought.

          Show
          Nicolas Lalevée added a comment - To properly fix the classpath issues, a simple fix will be needed in Ivy: allow to set the classloader used by Ivy to instanciate the class specified in the ivysettings.xml. But even if we impelemnt that fix, we are not sure to include the WorkspaceResolver in IvyDE as it force you to modify your settings. See previous comments. But you might be interested in IVYDE-89 , which suggest a simpler solution. I did not tested it thought.
          Hide
          Matt Goldspink added a comment -

          I've tried the new code that Nicolas submitted but it does not work because of classloader issues in Eclipse. The problem being that Ivy is loaded in a separate classloader to ivyde and is not parented to it so it can not see the WorkspaceResolver. I managed to hack it get it to work in my own eclipse by reverting IvyDE to using a bundled version of Ivy in Eclipse. This then made me think is it worth allowing separate versions of Ivy to be used on a per-project basis and have a default version bundled?

          Is there any news on whether the WorkspaceResolver will be bundled in IvyDE? My team has made a lot of use with when people work on interdependent projects to make sure we don't cause any api breaks and its solved a lot of headaches.

          Show
          Matt Goldspink added a comment - I've tried the new code that Nicolas submitted but it does not work because of classloader issues in Eclipse. The problem being that Ivy is loaded in a separate classloader to ivyde and is not parented to it so it can not see the WorkspaceResolver. I managed to hack it get it to work in my own eclipse by reverting IvyDE to using a bundled version of Ivy in Eclipse. This then made me think is it worth allowing separate versions of Ivy to be used on a per-project basis and have a default version bundled? Is there any news on whether the WorkspaceResolver will be bundled in IvyDE? My team has made a lot of use with when people work on interdependent projects to make sure we don't cause any api breaks and its solved a lot of headaches.
          Nicolas Lalevée made changes -
          Field Original Value New Value
          Attachment IVYDE-43-r641334.patch [ 12378634 ]
          Hide
          Nicolas Lalevée added a comment -

          In order to not loose Eugene code, here is an updated version of the patch. I didn't tested thought. But I fear some classpath issues in the suggested solution since Ivy and IvyDE are two seperated bundles. Ivy would instanciate the WorkspaceResolver whereas it actually is in IvyDE.

          Show
          Nicolas Lalevée added a comment - In order to not loose Eugene code, here is an updated version of the patch. I didn't tested thought. But I fear some classpath issues in the suggested solution since Ivy and IvyDE are two seperated bundles. Ivy would instanciate the WorkspaceResolver whereas it actually is in IvyDE.
          Hide
          Xavier Hanin added a comment -

          For not re-implementing Ivy features, I agree, reusing is the best.

          For the moment, Ivy only has a matching mechanism with revisions, which doesn't take care of branches and extra attributes. Branches and extra attrbitutes are only considered if they are part of the pattern, in this case they are considered during the listing, and that's all. There's already been a discussion about that, and it may be good to be able to deal with branches during matching, but it would imply parsing multiple module descriptors during resolve, which means bad performance... But in this very special case it doesn't hurt, and matching branch for equality is quite simple.

          For the versions, the way to do is to obtain a VersionMatcher from the Ivy instance, and then ask the VersionMatcher if the versions matches, and that's all. This should be pretty easy to add, maybe only if a version is present in the ivy file.

          Finally, about the trouble you had to figuree out how things work, I agree, it will be part of the 1.5 version to try to make things more clean and clear. I will at least try to make responsibilities more obvious (at least more documented ), even if the resolve algorithm stay complex (because I think it is complex to deal with conflict management and configurations). But we'll discuss more about that on the ASF mailing list, once svn will be migrated.

          Show
          Xavier Hanin added a comment - For not re-implementing Ivy features, I agree, reusing is the best. For the moment, Ivy only has a matching mechanism with revisions, which doesn't take care of branches and extra attributes. Branches and extra attrbitutes are only considered if they are part of the pattern, in this case they are considered during the listing, and that's all. There's already been a discussion about that, and it may be good to be able to deal with branches during matching, but it would imply parsing multiple module descriptors during resolve, which means bad performance... But in this very special case it doesn't hurt, and matching branch for equality is quite simple. For the versions, the way to do is to obtain a VersionMatcher from the Ivy instance, and then ask the VersionMatcher if the versions matches, and that's all. This should be pretty easy to add, maybe only if a version is present in the ivy file. Finally, about the trouble you had to figuree out how things work, I agree, it will be part of the 1.5 version to try to make things more clean and clear. I will at least try to make responsibilities more obvious (at least more documented ), even if the resolve algorithm stay complex (because I think it is complex to deal with conflict management and configurations). But we'll discuss more about that on the ASF mailing list, once svn will be migrated.
          Hide
          Xavier Hanin added a comment -

          As far as I understand the name of the dependency is matched against the ivy file corresponding to an ivy classpath container in the other eclipse project.

          To handle multiple versions and match only latest.integration, maybe we could have a setting saying:

          • match whatever the version in the ivy file is (current behavior)
          • match only when it actually matches the project ivy file version
            In this second case normal matching version would be used, considering the version of the dependency project to be :
          • the version in the ivy file if any
          • a default revision (a timestamp for example, or maybe something configurable, or the generated version used in the current patch) if there is no revision in the ivy file

          In this case we could distinguish between branches with some little extra work, and distinguish between version streams with no extra work except the setting of a revision in the working ivy file for the project dependency.

          Just a thought.

          Show
          Xavier Hanin added a comment - As far as I understand the name of the dependency is matched against the ivy file corresponding to an ivy classpath container in the other eclipse project. To handle multiple versions and match only latest.integration, maybe we could have a setting saying: match whatever the version in the ivy file is (current behavior) match only when it actually matches the project ivy file version In this second case normal matching version would be used, considering the version of the dependency project to be : the version in the ivy file if any a default revision (a timestamp for example, or maybe something configurable, or the generated version used in the current patch) if there is no revision in the ivy file In this case we could distinguish between branches with some little extra work, and distinguish between version streams with no extra work except the setting of a revision in the working ivy file for the project dependency. Just a thought.
          Hide
          Maarten Coene added a comment -

          Hi,

          great contribution!! You have my vote

          some remarks:

          • I also don't like that the ivy-configuration has to be modified to make this feature work. It would be great if that could be solved!
          • How do you match the name of the dependency to the eclipse project? Are you matching the name of the dependency to the name of the project? Or do you match the name of the dependency to the name of the module in the ivy.xml file of the other eclipse project (if it has an ivy.xml file) ?
          • It should be possible to define in more detail which dependencies are matched to eclipse projects. For instance, in my situation, I would like to use only the workspaceresolver if the revision of my dependency is "latest.integration".
          • It should be even more great if you could make multiple versions of the same projects in the same workspace to work (for instance by matching some attributes in the projects ivy.xml with the attributes of the dependency?)

          Maarten

          Show
          Maarten Coene added a comment - Hi, great contribution!! You have my vote some remarks: I also don't like that the ivy-configuration has to be modified to make this feature work. It would be great if that could be solved! How do you match the name of the dependency to the eclipse project? Are you matching the name of the dependency to the name of the project? Or do you match the name of the dependency to the name of the module in the ivy.xml file of the other eclipse project (if it has an ivy.xml file) ? It should be possible to define in more detail which dependencies are matched to eclipse projects. For instance, in my situation, I would like to use only the workspaceresolver if the revision of my dependency is "latest.integration". It should be even more great if you could make multiple versions of the same projects in the same workspace to work (for instance by matching some attributes in the projects ivy.xml with the attributes of the dependency?) Maarten
          Hide
          Xavier Hanin added a comment -

          Excellent! Terrific! Amazing!

          I was also dreaming of such a feature, and I know other users were waiting for it too. So it's great you did this wonderful work!

          I haven't tested so far, but if it works for you it should be ok for me. One question: with which eclipse version did you test it?

          Another thing, the only thing I don't really like is that you have to modify your ivy settings to make your feature work. Maybe we could find a way to make it work without that.
          Thinking aloud...
          I was thinking about a change which should not be too difficult... We could try to use the dictator resolver feature of Ivy: when you call resolve you can set a dictator resolver just before the call, and Ivy will ask to resolve everything with it instead of the normal settings. So we should be able to set the workspace resolver as being a dictator resolver. The only problem would be to go back to normal setting when the workspace resolver doesn't find what it is looking for. In this case we have no way with current Ivy version to ask for the real setting, except by setting the dictator resolver back to null just before asking:
          // WorkspaceResolver snippet
          ivy.setDictatorResolver(null);
          try

          { return ivy.getResolver(mid).getDependency(dd, data); }

          finally

          { ivy.setDictatorResolver(this); }

          This is not a very beautiful solution, but it should work.

          Maybe another solution is to modify the Ivy settings on the fly, after the call to configure, to add the workspace resolver (as you recommend to do by hand) and define it (or an upper chain, if we don't implement the delegation system in the workspace resolver itself) as a new default resolver.

          Ok, I see no really clean solution for the moment, but at least I think there are solutions which would make the feature even easier to use: no particular setup required! Just one thing in this case, add an option to disable the feature for all projects or for one project, so that we are sure it doesn't hurt.

          What do you think?

          Show
          Xavier Hanin added a comment - Excellent! Terrific! Amazing! I was also dreaming of such a feature, and I know other users were waiting for it too. So it's great you did this wonderful work! I haven't tested so far, but if it works for you it should be ok for me. One question: with which eclipse version did you test it? Another thing, the only thing I don't really like is that you have to modify your ivy settings to make your feature work. Maybe we could find a way to make it work without that. Thinking aloud... I was thinking about a change which should not be too difficult... We could try to use the dictator resolver feature of Ivy: when you call resolve you can set a dictator resolver just before the call, and Ivy will ask to resolve everything with it instead of the normal settings. So we should be able to set the workspace resolver as being a dictator resolver. The only problem would be to go back to normal setting when the workspace resolver doesn't find what it is looking for. In this case we have no way with current Ivy version to ask for the real setting, except by setting the dictator resolver back to null just before asking: // WorkspaceResolver snippet ivy.setDictatorResolver(null); try { return ivy.getResolver(mid).getDependency(dd, data); } finally { ivy.setDictatorResolver(this); } This is not a very beautiful solution, but it should work. Maybe another solution is to modify the Ivy settings on the fly, after the call to configure, to add the workspace resolver (as you recommend to do by hand) and define it (or an upper chain, if we don't implement the delegation system in the workspace resolver itself) as a new default resolver. Ok, I see no really clean solution for the moment, but at least I think there are solutions which would make the feature even easier to use: no particular setup required! Just one thing in this case, add an option to disable the feature for all projects or for one project, so that we are sure it doesn't hurt. What do you think?
          Hide
          Eugene Goldfarb added a comment -

          Thank you for encouraging comments and votes of confidence!

          Some thoughts about your thoughts:

          Xavier:

          • I mostly tested with Eclipse 3.2.1 and some testing with 3.1.1. I don't think I used any new APIs so it should work in any Eclipse version.
          • The dictator resolver seemed to me a great idea at first, and I started with it, but it was the first time I looked at Ivy code and it ended up not working since I was just setting up a chain including my workspace resolver followed by the default resolver from Ivy as the dictator. It didn't work since the default resolver was not always supposed to be used for every dependency. Same thing happened with replacing the default resolver - Ivy would completely miss my resolver since it used a different chain we had configured for our company's internal modules with the module tag which was not the default chain. Now that I see your suggestion of quickly replacing the dictator every time I can give it a try.

          I just found that configuring the resolver explicitly gives much more control over which modules and chains it should be used for. I see the case for it working out-of-the-box, so I'll think a little more about it. Definitely can do it when the ivyconf location is not set and it falls back to the default configuration.

          Maarten:
          I think Xavier already explained it - I find all projects that are open and have an Ivy container, and grab the ivy.xml being used to find out all the details about the module - its configurations, etc. I don't just match the module name but the whole ModuleId - organization and name. But NOT the ModuleRevisionId - revision, branch, extra attributes.

          I took a shortcut with the versions. First of all, at the company I work we don't have module revisions inside the working ivy files. They are determined at publish time. My resolver manufactures a revision by matching the revision that is being searched and appending the current timestamp to it, trying to make sure Ivy will pick it over other revisions coming from other chains - we have local, staging, and released repositories for internal modules. I cannot completely control Ivy, so I just try to trick it. I also had some trouble with dynamic revisions.
          But I guess sometimes the version would exist and my resolver could make use of it. I think it could be even more complicated if there are different latest strategies configured - I didn't test with those.

          • For your request of only using latest.integration, I'll try Xavier's suggestions from his last comment. I'm just afraid I'll end up re-implementing Ivy within my resolver to allow for all the various extra attributes, revision patterns, etc. I would like to use as many Ivy APIs as possible so it could do the work for me, but I just don't know everything it can do to help with this, like generating a matching revision based on latest strategy or file pattern, etc.
          • For multiple versions of the same project, I would have to see how I could match branch and revision, which are special attributes, and maybe even extra attributes now available with 1.4. I guess I need more test cases, I really tested with my specific needs and thought a little about other situations in general.

          In general, Ivy is a great product, but it was very hard to figure out what is going on. A couple of long nights with the debugger helped, but then the resolve semantics seemed to change when I upgraded to ivy 1.4. Before, I would return only the configuration the resolver was asking for. If a different configuration was needed later, another resolve would be done. Now, I had to return all possible configurations to make Ivy happy since it could later decide in needed another configuration. I actually may need an improvement there and only return the non-private configurations.

          • One more thing I didn't like with IvyDE is that modules are not alphabetized when they are put into the Ivy container, so it is very hard to find if a module got resolved or not. I think it would be an easy change, probably another JIRA issue.
          • Eugene
          Show
          Eugene Goldfarb added a comment - Thank you for encouraging comments and votes of confidence! Some thoughts about your thoughts: Xavier: I mostly tested with Eclipse 3.2.1 and some testing with 3.1.1. I don't think I used any new APIs so it should work in any Eclipse version. The dictator resolver seemed to me a great idea at first, and I started with it, but it was the first time I looked at Ivy code and it ended up not working since I was just setting up a chain including my workspace resolver followed by the default resolver from Ivy as the dictator. It didn't work since the default resolver was not always supposed to be used for every dependency. Same thing happened with replacing the default resolver - Ivy would completely miss my resolver since it used a different chain we had configured for our company's internal modules with the module tag which was not the default chain. Now that I see your suggestion of quickly replacing the dictator every time I can give it a try. I just found that configuring the resolver explicitly gives much more control over which modules and chains it should be used for. I see the case for it working out-of-the-box, so I'll think a little more about it. Definitely can do it when the ivyconf location is not set and it falls back to the default configuration. Maarten: I think Xavier already explained it - I find all projects that are open and have an Ivy container, and grab the ivy.xml being used to find out all the details about the module - its configurations, etc. I don't just match the module name but the whole ModuleId - organization and name. But NOT the ModuleRevisionId - revision, branch, extra attributes. I took a shortcut with the versions. First of all, at the company I work we don't have module revisions inside the working ivy files. They are determined at publish time. My resolver manufactures a revision by matching the revision that is being searched and appending the current timestamp to it, trying to make sure Ivy will pick it over other revisions coming from other chains - we have local, staging, and released repositories for internal modules. I cannot completely control Ivy, so I just try to trick it. I also had some trouble with dynamic revisions. But I guess sometimes the version would exist and my resolver could make use of it. I think it could be even more complicated if there are different latest strategies configured - I didn't test with those. For your request of only using latest.integration, I'll try Xavier's suggestions from his last comment. I'm just afraid I'll end up re-implementing Ivy within my resolver to allow for all the various extra attributes, revision patterns, etc. I would like to use as many Ivy APIs as possible so it could do the work for me, but I just don't know everything it can do to help with this, like generating a matching revision based on latest strategy or file pattern, etc. For multiple versions of the same project, I would have to see how I could match branch and revision, which are special attributes, and maybe even extra attributes now available with 1.4. I guess I need more test cases, I really tested with my specific needs and thought a little about other situations in general. In general, Ivy is a great product, but it was very hard to figure out what is going on. A couple of long nights with the debugger helped, but then the resolve semantics seemed to change when I upgraded to ivy 1.4. Before, I would return only the configuration the resolver was asking for. If a different configuration was needed later, another resolve would be done. Now, I had to return all possible configurations to make Ivy happy since it could later decide in needed another configuration. I actually may need an improvement there and only return the non-private configurations. One more thing I didn't like with IvyDE is that modules are not alphabetized when they are put into the Ivy container, so it is very hard to find if a module got resolved or not. I think it would be an easy change, probably another JIRA issue. Eugene
          Hide
          Eugene Goldfarb added a comment -

          Attached screenshot showing how projects can be linked. The top projects depends on the other two projects (misc-services and core). misc-services also depends on core.

          Show
          Eugene Goldfarb added a comment - Attached screenshot showing how projects can be linked. The top projects depends on the other two projects (misc-services and core). misc-services also depends on core.
          Eugene Goldfarb created issue -

            People

            • Assignee:
              Unassigned
              Reporter:
              Eugene Goldfarb
            • Votes:
              5 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development