Uploaded image for project: 'OFBiz'
  1. OFBiz
  2. OFBIZ-9182

Create a separate svn repository for OFBiz official plugins

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: Upcoming Release
    • Fix Version/s: None
    • Component/s: None

      Description

      This issue is related to the discussion found in this thread in which the community approved restructuring our repositories. To achieve this task the following needs to be done (in this order)

      1. Update the gradle scripts to assume that no plugins exist in the plugins directory by default and no component-load.xml exists. It should follow the same logic in loading the components as found in the ComponentContainer class. Also the activation and deactivation of plugins happens in ofbiz-component.xml, not in component-load.xml
      2. Add a new task to gradle called pullPluginSource that retrieves a plugin from subversion and defaults to the official plugins repository of Apache OFBiz. This task mostly serve "Trunk" because it always needs the latest source code of the plugins.
      3. delete plugins/component-load.xml
      4. move all plugins to a new repository called http://svn.apache.org/repos/asf/ofbiz/ofbiz-plugins
      5. move the core framework to a new repository called http://svn.apache.org/repos/asf/ofbiz/ofbiz-framework
      6. fix buildbot to point to the new framework location
      7. update documentation where applicable including README.md
      1. OFBIZ-9182-subversion-embedded.patch
        2 kB
        Taher Alkhateeb
      2. pluginsList.txt
        0.2 kB
        Jacques Le Roux
      3. pullAllPluginsSource.log
        64 kB
        Jacques Le Roux

        Issue Links

          Activity

          Hide
          taher Taher Alkhateeb added a comment -

          First commit delivered in r1780790 which enables plugins to load without component-load.xml. Most of the work went into the build scripts.

          The next step is to create "pullPluginSource" gradle task to handle pulling plugins into the code base.

          Show
          taher Taher Alkhateeb added a comment - First commit delivered in r1780790 which enables plugins to load without component-load.xml. Most of the work went into the build scripts. The next step is to create "pullPluginSource" gradle task to handle pulling plugins into the code base.
          Hide
          pfm.smits Pierre Smits added a comment -

          So, to understand the elements of the description correctly:

          #1 is utilising the logic (code) used for components/applications in the hot-deploy folder to start these components and applications there, and apply that logic to the into plugins renamed special purpose folder?
          #2 is the pullPluginSource task basically a copy/paste action similar adopters have to perform to get components/applications into the hot-deploy folder?
          #3 is basically removing the component-load.xml form the renamed special purpose folder making the plugin folders behaving the same as the hot-deploy folder?

          Would it therefore not have been more prudent, and effort and time saving, to think through the concepts of this whole 'plugin' approach more thoroughly than to have this community go through al lot of confusion and conflict to arrive to at what basically is:
          #1 the hot-deploy folder for special purpose components and applications;
          #2 moving components/applications in the now renamed special purpose folder out of the OFBiz trunk, future release branches and releases and into attic?

          Show
          pfm.smits Pierre Smits added a comment - So, to understand the elements of the description correctly: #1 is utilising the logic (code) used for components/applications in the hot-deploy folder to start these components and applications there, and apply that logic to the into plugins renamed special purpose folder? #2 is the pullPluginSource task basically a copy/paste action similar adopters have to perform to get components/applications into the hot-deploy folder? #3 is basically removing the component-load.xml form the renamed special purpose folder making the plugin folders behaving the same as the hot-deploy folder? Would it therefore not have been more prudent, and effort and time saving, to think through the concepts of this whole 'plugin' approach more thoroughly than to have this community go through al lot of confusion and conflict to arrive to at what basically is: #1 the hot-deploy folder for special purpose components and applications; #2 moving components/applications in the now renamed special purpose folder out of the OFBiz trunk, future release branches and releases and into attic?
          Hide
          taher Taher Alkhateeb added a comment -

          removed activate / deactivate plugin in r1780828.

          Show
          taher Taher Alkhateeb added a comment - removed activate / deactivate plugin in r1780828.
          Hide
          taher Taher Alkhateeb added a comment -

          There are two ways to pull a plugin from source: Either have subversion installed on the machine and use exec

          {...}

          OR use an embedded subversion based on Java and hence avoid dependence on environment.

          Attached is a solution that is independent of the environment, it pulls extra dependencies which is why I share here before committing to see if we can improve this.

          Show
          taher Taher Alkhateeb added a comment - There are two ways to pull a plugin from source: Either have subversion installed on the machine and use exec {...} OR use an embedded subversion based on Java and hence avoid dependence on environment. Attached is a solution that is independent of the environment, it pulls extra dependencies which is why I share here before committing to see if we can improve this.
          Hide
          soledad Nicolas Malin added a comment -

          I mostly agree with you work.
          Just a remark on

          Add a new task to gradle called pullPluginSource that retrieves a plugin from subversion and defaults to the official plugins repository of Apache OFBiz. This task mostly server "Trunk" because it always needs the latest source code of the plugins.

          for my point of view, this point isn't correct.
          We need to resolve the plugin source code from the head of the branch who he present, because we can have small incompatibility between a stable branch and the trunk.
          I propose to add the ofbiz version/branch on a property file (or an other best way) to indicate on ofbiz what the version is related.
          With this a user can use a future published version like 16.10.01 call the plugin source to pull the project component and automatically OFBiz can resolve the plugin from http://svn.apache.org/repos/asf/ofbiz/ofbiz-plugins/release16.10/projectmgr

          Perhaps my suggest is in advance on your plan or I mistake something

          Show
          soledad Nicolas Malin added a comment - I mostly agree with you work. Just a remark on Add a new task to gradle called pullPluginSource that retrieves a plugin from subversion and defaults to the official plugins repository of Apache OFBiz. This task mostly server "Trunk" because it always needs the latest source code of the plugins. for my point of view, this point isn't correct. We need to resolve the plugin source code from the head of the branch who he present, because we can have small incompatibility between a stable branch and the trunk. I propose to add the ofbiz version/branch on a property file (or an other best way) to indicate on ofbiz what the version is related. With this a user can use a future published version like 16.10.01 call the plugin source to pull the project component and automatically OFBiz can resolve the plugin from http://svn.apache.org/repos/asf/ofbiz/ofbiz-plugins/release16.10/projectmgr Perhaps my suggest is in advance on your plan or I mistake something
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Taher, I reviewed r1780790 and it's OK with me (not tested yet).

          About your proposed patch, it also looks good to me (though not tested). Something though which seems to miss in gradle-svntools-plugin is the svn:externals feature. It would be great if we could not only check plugins out, but also keep them updated later...

          Before commenting on Nicolas's comment, I'm not sure I clearly understand the sentence

          This task mostly server "Trunk" because it always needs the latest source code of the plugins.

          Now if I understand it the same way than Nicolas (ie plugins are always based on trunk) then I agree with Nicolas, and BTW

          small incompatibility between a stable branch and the trunk.

          is even an euphemism

          Nicolas, when you say

          With this a user can use a future published version like 16.10.01

          I guess you mean 16.10.02? Because 16.10.01 is already released.

          If we really are on the same page, I think that rather than using a Java properties file, we should rely on the svn:external svn property.

          Show
          jacques.le.roux Jacques Le Roux added a comment - Taher, I reviewed r1780790 and it's OK with me (not tested yet). About your proposed patch, it also looks good to me (though not tested). Something though which seems to miss in gradle-svntools-plugin is the svn:externals feature. It would be great if we could not only check plugins out, but also keep them updated later... Before commenting on Nicolas's comment, I'm not sure I clearly understand the sentence This task mostly server "Trunk" because it always needs the latest source code of the plugins. Now if I understand it the same way than Nicolas (ie plugins are always based on trunk) then I agree with Nicolas, and BTW small incompatibility between a stable branch and the trunk. is even an euphemism Nicolas, when you say With this a user can use a future published version like 16.10.01 I guess you mean 16.10.02? Because 16.10.01 is already released. If we really are on the same page, I think that rather than using a Java properties file, we should rely on the svn:external svn property.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          I tested r1780790, works for me, thanks Taher!

          Show
          jacques.le.roux Jacques Le Roux added a comment - I tested r1780790, works for me, thanks Taher!
          Hide
          taher Taher Alkhateeb added a comment -

          Hello Nicolas and Jacques,

          I will answer both your questions / comments here since they are related. This is my proposition for how the plugin mechanism works (nothing concrete yet, just ideas):

          • All OFBiz future releases would have equivalent plugin releases. So for example, if we have say .. release 18.04, we can have a plugin with the same version that is guaranteed to be compatible. You download the plugin using the ./gradlew pullPlugin, and NOT the ./gradlew pullPluginSource
          • OFBiz trunk, on the other hand, is not a released code. So, in order to install and test plugins for Trunk, you use the ./gradlew pullPluginSource

          So published plugins exist only for releases (NOT for trunk). In my opinion this would greatly simplify the release process and it would give us independence from the underlying source code revision system.

          In other words, it does not make a lot of sense to go down to revision-by-revision tracking of what works with that, that's just too much work. Instead, it's much easier to just have releases (with matching numbers) between released framework and plugins, and to have source-code to source-code compatibility between the ofbiz source code and the plugins source code.

          So the next question is, how do we make sure that trunk always works with plugins? Well, the simplest answer that I can come up with is automated builds. We can have one buildbot script for OFBiz, and another buildbot script for Plugins. This way we can always catch problems while maintaining separate repositories.

          So these are my thoughts, you're welcome to modify or start discussing on ML for more

          Show
          taher Taher Alkhateeb added a comment - Hello Nicolas and Jacques, I will answer both your questions / comments here since they are related. This is my proposition for how the plugin mechanism works (nothing concrete yet, just ideas): All OFBiz future releases would have equivalent plugin releases. So for example, if we have say .. release 18.04, we can have a plugin with the same version that is guaranteed to be compatible. You download the plugin using the ./gradlew pullPlugin, and NOT the ./gradlew pullPluginSource OFBiz trunk, on the other hand, is not a released code. So, in order to install and test plugins for Trunk, you use the ./gradlew pullPluginSource So published plugins exist only for releases (NOT for trunk). In my opinion this would greatly simplify the release process and it would give us independence from the underlying source code revision system. In other words, it does not make a lot of sense to go down to revision-by-revision tracking of what works with that, that's just too much work. Instead, it's much easier to just have releases (with matching numbers) between released framework and plugins, and to have source-code to source-code compatibility between the ofbiz source code and the plugins source code. So the next question is, how do we make sure that trunk always works with plugins? Well, the simplest answer that I can come up with is automated builds. We can have one buildbot script for OFBiz, and another buildbot script for Plugins. This way we can always catch problems while maintaining separate repositories. So these are my thoughts, you're welcome to modify or start discussing on ML for more
          Hide
          jacques.le.roux Jacques Le Roux added a comment - - edited

          Hi Taher,

          Just thinking out loud for now...

          We don't need 2 Buildbot scripts if we can use svn externals http://svnbook.red-bean.com/en/1.8/svn.advanced.externals.html
          Note about automation: currently https://github.com/martoe/gradle-svntools-plugin does not support svn propedit, we could contribute, it's groovy code after all...
          BTW, there is a similar solutions with Git if we need it later https://stackoverflow.com/questions/571232/svnexternals-equivalent-in-git

          I have one worry, what about people relying on branches (evolving with new features eg R16.11) and not releases (fixed but bugs, eg 16.11.01) ? It seems they will loose something, don't they? Fortunately svn externals allows to define specific revisions. For instance we could define a revision for each plugin. As you said maybe too much, but maybe also convenient. Anyway this needs at least to be evaluated (ie more than just thinking out loud late in the evening...)

          Show
          jacques.le.roux Jacques Le Roux added a comment - - edited Hi Taher, Just thinking out loud for now... We don't need 2 Buildbot scripts if we can use svn externals http://svnbook.red-bean.com/en/1.8/svn.advanced.externals.html Note about automation: currently https://github.com/martoe/gradle-svntools-plugin does not support svn propedit, we could contribute, it's groovy code after all... BTW, there is a similar solutions with Git if we need it later https://stackoverflow.com/questions/571232/svnexternals-equivalent-in-git I have one worry, what about people relying on branches (evolving with new features eg R16.11) and not releases (fixed but bugs, eg 16.11.01) ? It seems they will loose something, don't they? Fortunately svn externals allows to define specific revisions. For instance we could define a revision for each plugin. As you said maybe too much, but maybe also convenient. Anyway this needs at least to be evaluated (ie more than just thinking out loud late in the evening...)
          Hide
          soledad Nicolas Malin added a comment -

          Taher Alkhateeb Yes I understand well your priority of simplicity. What I tried to explain without success that if a plugin is published on a stable branch, you can on different time need to pull it from the svn src. But in fact, maybe it's not a problem (I just sharing my brain ^^).

          Jacques Le Roux Resolve the stable branch from the svn implies that it would be unusable from publish ofbiz version, but only work from a source checkout. But in fact, maybe it's not a problem

          Show
          soledad Nicolas Malin added a comment - Taher Alkhateeb Yes I understand well your priority of simplicity. What I tried to explain without success that if a plugin is published on a stable branch, you can on different time need to pull it from the svn src. But in fact, maybe it's not a problem (I just sharing my brain ^^). Jacques Le Roux Resolve the stable branch from the svn implies that it would be unusable from publish ofbiz version, but only work from a source checkout. But in fact, maybe it's not a problem
          Hide
          taher Taher Alkhateeb added a comment -

          Hi Nicolas, What would be the purpose of getting the latest plugin form version control to a release?

          Show
          taher Taher Alkhateeb added a comment - Hi Nicolas, What would be the purpose of getting the latest plugin form version control to a release?
          Hide
          jacques.le.roux Jacques Le Roux added a comment - - edited

          Nicolas,

          Has Taher suggested, people using releases would have available corresponding plugins released. They don't need to use pullPluginSource only pullPlugin

          In other cases (relying on checked out sources), svn externals is perfect. Because it opens all possibilities svn provides. When associating a plugin, you can either use a revision or w/o revision which means taking HEAD.

          We though miss the automation. Because so far the Gradle plugin has no task for properties. Properties are very powerful features of Svn, often neglected.

          Disclaimer: I must say I have not yet a clear picture in mind, just ideas...

          Show
          jacques.le.roux Jacques Le Roux added a comment - - edited Nicolas, Has Taher suggested, people using releases would have available corresponding plugins released. They don't need to use pullPluginSource only pullPlugin In other cases (relying on checked out sources), svn externals is perfect. Because it opens all possibilities svn provides. When associating a plugin, you can either use a revision or w/o revision which means taking HEAD. We though miss the automation. Because so far the Gradle plugin has no task for properties. Properties are very powerful features of Svn, often neglected. Disclaimer: I must say I have not yet a clear picture in mind, just ideas...
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          OK, it clear to me now. We don't need pullPluginSource if we set a svn external for the plugins repo.

          Taher said

          There are two ways to pull a plugin from source: Either have subversion installed on the machine and use exec

          Unknown macro: {...}

          OR use an embedded subversion based on Java and hence avoid dependence on environment.

          Actually people should not have to worry about using "exec" manually. That's what a svn external for the plugins repo would do automatically for them, checking out the whole plugins dir without any efforts and even noticing it.

          I recap:

          1. People using released packages use pullPlugin to get the released plugin/s they want (and more because of plugins interdependencies)
          2. People not using released packages, ie using checked out working copies, don't have to worry they get all plugins with the svn externals as it is now. The dependencies will be resolved while building as it is now

          But then for svn externals, what for people who want to use only one or few plugins? Then maybe we need to reconsider component.load in plugins dir and de/activatePlugin tasks...

          So the decision about using svn external or pullPluginSource is up to the community.

          Maybe pullPluginSource is clearer because you easily get only what you want, but what about plugins inter-dependencies? Also if you want a lot of plugins it's kinda a pain. For now we have "only" 19 plugins but what's next?

          To summarize, as long as plugins inter-dependencies will not be resolved svn externals is more convenient.

          About resolving plugins inter-dependencies, is it even possible (we finally let it be in applications)? Consider DLL and Jar hells, even Microsoft and Sun were not able to cleanly do it. And now you have Java 9 and its modules coming: http://blog.codefx.org/java/dev/will-there-be-module-hell/

          Show
          jacques.le.roux Jacques Le Roux added a comment - OK, it clear to me now. We don't need pullPluginSource if we set a svn external for the plugins repo. Taher said There are two ways to pull a plugin from source: Either have subversion installed on the machine and use exec Unknown macro: {...} OR use an embedded subversion based on Java and hence avoid dependence on environment. Actually people should not have to worry about using "exec" manually. That's what a svn external for the plugins repo would do automatically for them, checking out the whole plugins dir without any efforts and even noticing it. I recap: People using released packages use pullPlugin to get the released plugin/s they want (and more because of plugins interdependencies) People not using released packages, ie using checked out working copies, don't have to worry they get all plugins with the svn externals as it is now. The dependencies will be resolved while building as it is now But then for svn externals, what for people who want to use only one or few plugins? Then maybe we need to reconsider component.load in plugins dir and de/activatePlugin tasks... So the decision about using svn external or pullPluginSource is up to the community. Maybe pullPluginSource is clearer because you easily get only what you want, but what about plugins inter-dependencies? Also if you want a lot of plugins it's kinda a pain. For now we have "only" 19 plugins but what's next? To summarize, as long as plugins inter-dependencies will not be resolved svn externals is more convenient. About resolving plugins inter-dependencies, is it even possible (we finally let it be in applications)? Consider DLL and Jar hells, even Microsoft and Sun were not able to cleanly do it. And now you have Java 9 and its modules coming: http://blog.codefx.org/java/dev/will-there-be-module-hell/
          Hide
          taher Taher Alkhateeb added a comment -

          I don't think you got what I said about the two ways to pull (externals is not related here). I'll try to explain some more the benefits of the approach in the patch.

          • First of all, the pullPluginSource won't stop you from using subversion if you want to. It's just a convenience task
          • In the first implementation way I described above, you need svn installed, in the second you don't
          • also using my patch we do not rely on a specific version control system. We can change it by changing implementation of pullPluginSource
          • with pullPluginSource we can easily add the logic to pull dependencies which we cannot do with raw subversion. The implementation is already there in "pullPlugin".
          • activating / deactivating plugins (or any component) can be done using ofbiz-component.xml. I don't understand why component-load.xml is needed in your argument above.

          Anyway, If you want to change the implementation then open this discussion in the ML

          Show
          taher Taher Alkhateeb added a comment - I don't think you got what I said about the two ways to pull (externals is not related here). I'll try to explain some more the benefits of the approach in the patch. First of all, the pullPluginSource won't stop you from using subversion if you want to. It's just a convenience task In the first implementation way I described above, you need svn installed, in the second you don't also using my patch we do not rely on a specific version control system. We can change it by changing implementation of pullPluginSource with pullPluginSource we can easily add the logic to pull dependencies which we cannot do with raw subversion. The implementation is already there in "pullPlugin". activating / deactivating plugins (or any component) can be done using ofbiz-component.xml. I don't understand why component-load.xml is needed in your argument above. Anyway, If you want to change the implementation then open this discussion in the ML
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Actually I very well got what you said. I'm just adding that using externals is a very simple solution that should be considered for people who are using svn to check out things.

          * First of all, the pullPluginSource won't stop you from using subversion if you want to. It's just a convenience task

          Yes got that

          * In the first implementation way I described above, you need svn installed, in the second you don't

          Yes got that too, as I said what I'm suggesting is for people using svn to "automatically" have the same that we have now. For instance for committers it's very convenient.

          * also using my patch we do not rely on a specific version control system. We can change it by changing implementation of pullPluginSource

          That could be indeed interesting if we move to Git for instance. Note that, as I said above, the same than svn externals can also be achieved by Git.

          * with pullPluginSource we can easily add the logic to pull dependencies which we cannot do with raw subversion. The implementation is already there in "pullPlugin".

          OK good, not needed if we use svn externals for OOTB plugins. It could interesting if pullPluginSource accepted an url parameter for people having sources in other repos. This is what I was looking for when I spoke about Jitpack.

          * activating / deactivating plugins (or any component) can be done using ofbiz-component.xml. I don't understand why component-load.xml is needed in your argument above.

          Yes right, that's a very good point I missed. Indeed using the enabled attribute of the ofbiz-component element is the way.

          So I think we can agree that having both solutions is possible. For svn externals, after changing the repo structure, it's only a matter of adding the property, et voilĂ . Do you think we need to discuss that on dev ML?

          Show
          jacques.le.roux Jacques Le Roux added a comment - Actually I very well got what you said. I'm just adding that using externals is a very simple solution that should be considered for people who are using svn to check out things. * First of all, the pullPluginSource won't stop you from using subversion if you want to. It's just a convenience task Yes got that * In the first implementation way I described above, you need svn installed, in the second you don't Yes got that too, as I said what I'm suggesting is for people using svn to "automatically" have the same that we have now. For instance for committers it's very convenient. * also using my patch we do not rely on a specific version control system. We can change it by changing implementation of pullPluginSource That could be indeed interesting if we move to Git for instance. Note that, as I said above, the same than svn externals can also be achieved by Git. * with pullPluginSource we can easily add the logic to pull dependencies which we cannot do with raw subversion. The implementation is already there in "pullPlugin". OK good, not needed if we use svn externals for OOTB plugins. It could interesting if pullPluginSource accepted an url parameter for people having sources in other repos. This is what I was looking for when I spoke about Jitpack. * activating / deactivating plugins (or any component) can be done using ofbiz-component.xml. I don't understand why component-load.xml is needed in your argument above. Yes right, that's a very good point I missed. Indeed using the enabled attribute of the ofbiz-component element is the way. So I think we can agree that having both solutions is possible. For svn externals, after changing the repo structure, it's only a matter of adding the property, et voilĂ . Do you think we need to discuss that on dev ML?
          Hide
          soledad Nicolas Malin added a comment -

          Ok Thanks for all details.

          I propose that Taher commit is improvement and we continue the svn migration because it's a simple step and with the feedback we can check if we need to continue the imrpovement or we can simply define so best pratice.

          Show
          soledad Nicolas Malin added a comment - Ok Thanks for all details. I propose that Taher commit is improvement and we continue the svn migration because it's a simple step and with the feedback we can check if we need to continue the imrpovement or we can simply define so best pratice.
          Hide
          taher Taher Alkhateeb added a comment -

          I agree with Nicolas, why introduce two solutions? Let's take it one step at a time. I see no benefit to confusing users with two ways of achieving the same thing.

          Show
          taher Taher Alkhateeb added a comment - I agree with Nicolas, why introduce two solutions? Let's take it one step at a time. I see no benefit to confusing users with two ways of achieving the same thing.
          Hide
          jacques.le.roux Jacques Le Roux added a comment - - edited

          Taher,

          Are you sure that's what Nicolas said? Nicolas?

          Anyway I agree we can go on, adding a svn externals is few seconds...

          Show
          jacques.le.roux Jacques Le Roux added a comment - - edited Taher, Are you sure that's what Nicolas said? Nicolas? Anyway I agree we can go on, adding a svn externals is few seconds...
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          To be more clear, we have 3 types of users

          1. committers
          2. users basing their develoment on working copies
          3. users basing their develoment on released package
          1. want svn:externals
          2. have svn:externals but want pullPluginSource to access their own repos
          3. want pullPlugin
          Show
          jacques.le.roux Jacques Le Roux added a comment - To be more clear, we have 3 types of users committers users basing their develoment on working copies users basing their develoment on released package want svn:externals have svn:externals but want pullPluginSource to access their own repos want pullPlugin
          Hide
          taher Taher Alkhateeb added a comment -

          all mentioned types of users above can use pullPluginSource. I'm missing the exact point of added value of of using a different solution to pull the source code in multiple different ways.

          Show
          taher Taher Alkhateeb added a comment - all mentioned types of users above can use pullPluginSource. I'm missing the exact point of added value of of using a different solution to pull the source code in multiple different ways.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Yes it's cumulative, the point is, as I already explained above, it's easier to have 19 plugins ready to use with svn:externals, when, if you want them with pullPluginSource (and have not svn:external), you have to use it 19 times, right?
          Also I guess adding an url, for users whot want pullPluginSource to access their own repos, is not a big deal?

          Show
          jacques.le.roux Jacques Le Roux added a comment - Yes it's cumulative, the point is, as I already explained above, it's easier to have 19 plugins ready to use with svn:externals, when, if you want them with pullPluginSource (and have not svn:external), you have to use it 19 times, right? Also I guess adding an url, for users whot want pullPluginSource to access their own repos, is not a big deal?
          Hide
          taher Taher Alkhateeb added a comment -

          We can have a task, for example, called pullPluginSourceAll to pull everything in the plugins repository. The reason I am making a recommendation to stick with gradle tasks as opposed to depending on the underlying version control system is to reduce complexity.

          Show
          taher Taher Alkhateeb added a comment - We can have a task, for example, called pullPluginSourceAll to pull everything in the plugins repository. The reason I am making a recommendation to stick with gradle tasks as opposed to depending on the underlying version control system is to reduce complexity.
          Hide
          jacques.le.roux Jacques Le Roux added a comment - - edited

          Complexity, that's also my point. With a pullPluginSourceAll you have to maintain it each time you change the content of the plugins folder. While setting svn:externals is done in seconds for "eternity". Committers, users don't need to use pullPluginSourceAll. The plugins are already there checked out with the rest (for now ofbiz-core, later another externals can be set for ofbiz-framework and ofbiz-core as Jacopo suggested, and what-not anyway). For users using working copies, if they don't want to use a plugin they can either deactivate it or simply get rid of it by deleting its folder.

          Show
          jacques.le.roux Jacques Le Roux added a comment - - edited Complexity, that's also my point. With a pullPluginSourceAll you have to maintain it each time you change the content of the plugins folder. While setting svn:externals is done in seconds for "eternity". Committers, users don't need to use pullPluginSourceAll. The plugins are already there checked out with the rest (for now ofbiz-core, later another externals can be set for ofbiz-framework and ofbiz-core as Jacopo suggested, and what-not anyway). For users using working copies, if they don't want to use a plugin they can either deactivate it or simply get rid of it by deleting its folder.
          Hide
          taher Taher Alkhateeb added a comment -

          Well, svn externals could be an implementation detail then embedded inside pullPluginSourceAll. Anway, I'm tired of going over this painful level of detail when we did not restructure yet. I would really rather postpone this discussion until we get the pieces working first, and I really think this discussion should go in the ML given its broad scope and impact

          Show
          taher Taher Alkhateeb added a comment - Well, svn externals could be an implementation detail then embedded inside pullPluginSourceAll. Anway, I'm tired of going over this painful level of detail when we did not restructure yet. I would really rather postpone this discussion until we get the pieces working first, and I really think this discussion should go in the ML given its broad scope and impact
          Hide
          jacques.le.roux Jacques Le Roux added a comment - - edited

          As I said already, the pb with gradle-svntools-plugin is it does not handle properties (yet?) so I don't see how at this stage. You said:

          svn externals could be an implementation detail then embedded inside pullPluginSourceAll

          Morevoer, as setting an svn:externals is only a matter of a property commit (done once), I don't see the need to have that embedded inside pullPluginSourceAll at all

          IMO pullPluginSource is only interesting if we can pass an URL for non ASF repos.

          But I agree we can go on with restructuring the OFBiz repo branch, and discuss further on dev ML if we should use svn:externals or not. I think we already put all the arguments on the table.

          Show
          jacques.le.roux Jacques Le Roux added a comment - - edited As I said already, the pb with gradle-svntools-plugin is it does not handle properties (yet?) so I don't see how at this stage. You said: svn externals could be an implementation detail then embedded inside pullPluginSourceAll Morevoer, as setting an svn:externals is only a matter of a property commit (done once), I don't see the need to have that embedded inside pullPluginSourceAll at all IMO pullPluginSource is only interesting if we can pass an URL for non ASF repos. But I agree we can go on with restructuring the OFBiz repo branch, and discuss further on dev ML if we should use svn:externals or not. I think we already put all the arguments on the table.
          Hide
          taher Taher Alkhateeb added a comment -

          Committed the implementation of pullPluginSource and also updated README.md in r1782605

          Show
          taher Taher Alkhateeb added a comment - Committed the implementation of pullPluginSource and also updated README.md in r1782605
          Hide
          deepak.dixit Deepak Dixit added a comment -

          One more question,

          Does this restructuring automatically reflect on https://github.com/apache/ofbiz or we need to ask INFRA team to update this on github as well?

          Show
          deepak.dixit Deepak Dixit added a comment - One more question, Does this restructuring automatically reflect on https://github.com/apache/ofbiz or we need to ask INFRA team to update this on github as well?
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          I'm not sure, at 90% I'd say it's automated (a mirror). But I think it's a good idea to ask on infra hipchat for best practices. Anyway this should not refrain us to restructure...

          Show
          jacques.le.roux Jacques Le Roux added a comment - I'm not sure, at 90% I'd say it's automated (a mirror). But I think it's a good idea to ask on infra hipchat for best practices. Anyway this should not refrain us to restructure...
          Hide
          deepak.dixit Deepak Dixit added a comment -

          Svn restructuring done at r#1782651 and r#1782652
          Created tag for r#1782650 https://svn.apache.org/repos/asf/ofbiz/tags/beforeSvnRestructuring

          Show
          deepak.dixit Deepak Dixit added a comment - Svn restructuring done at r#1782651 and r#1782652 Created tag for r#1782650 https://svn.apache.org/repos/asf/ofbiz/tags/beforeSvnRestructuring
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          I think we can close here. I let you do it Taher, thanks

          Show
          jacques.le.roux Jacques Le Roux added a comment - I think we can close here. I let you do it Taher, thanks
          Hide
          taher Taher Alkhateeb added a comment -

          Not yet done, we still have buildbot and making sure the API works correctly

          Show
          taher Taher Alkhateeb added a comment - Not yet done, we still have buildbot and making sure the API works correctly
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Since the title here is "Create a separate svn repository for OFBiz official plugins" and it's done, should we not create a new Jira for that and related?

          Show
          jacques.le.roux Jacques Le Roux added a comment - Since the title here is "Create a separate svn repository for OFBiz official plugins" and it's done, should we not create a new Jira for that and related?
          Hide
          taher Taher Alkhateeb added a comment -

          Read the description and steps described

          Show
          taher Taher Alkhateeb added a comment - Read the description and steps described
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          OK, BuildBot and documentation are on their way INFRA-13497. There is also an action needed for GitHub INFRA-13496

          Show
          jacques.le.roux Jacques Le Roux added a comment - OK, BuildBot and documentation are on their way INFRA-13497 . There is also an action needed for GitHub INFRA-13496
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          I think the documentation is OK, please check

          Show
          jacques.le.roux Jacques Le Roux added a comment - I think the documentation is OK, please check
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Should we not rename Eclipse project name to ofbiz-framework ?

          Show
          jacques.le.roux Jacques Le Roux added a comment - Should we not rename Eclipse project name to ofbiz-framework ?
          Hide
          taher Taher Alkhateeb added a comment -

          why for what purpose?

          Show
          taher Taher Alkhateeb added a comment - why for what purpose?
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          To clarify the difference with the previous one. Note that I'm just asking, actually it's fine with me as is.

          Show
          jacques.le.roux Jacques Le Roux added a comment - To clarify the difference with the previous one. Note that I'm just asking, actually it's fine with me as is.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          The Buildbot part is OK.

          Show
          jacques.le.roux Jacques Le Roux added a comment - The Buildbot part is OK.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          I have quickly tried to create a pullAllPluginsSource task adding the code below

          Index: build.gradle
          ===================================================================
          --- build.gradle	(revision 1783441)
          +++ build.gradle	(working copy)
          @@ -745,6 +745,17 @@
               }
           }
          
          +
          +task pullAllPluginsSource(group: ofbizPlugin, description: 'Download and install all plugins from source control') {
          +
          +    String[] pluginsList = new File('pluginsList.txt')
          +    for (plugin in pluginsList) {
          +        def svnUrl = "https://svn.apache.org/repos/asf/ofbiz/ofbiz-plugins/trunk/" + plugin
          +        def workspaceDir = "${rootDir}/plugins/" + plugin
          +        installPlugin(plugin)
          +    }
          +}
          +
           // ========== Clean up tasks ==========
           task cleanCatalina(group: cleanupGroup, description: 'Clean Catalina data in runtime/catalina/work') {
               doLast { delete "${rootDir}/runtime/catalina/work" }
          

          But I got a weird issue. The task stacks the call to installPlugin and so also stacks Gradle Daemons (I use it locally). So the memory increases fast and at some point you either get ouf or memory, or if you care before the whole thing crashes anyway. See pullAllPluginsSource.log and pluginsList.txt attached files.

          It would have been quite cool but it's obviously not good. What would be the right way?

          BTW "don't try that at home" In a last attempt, I rebooted, cleared all possible memory and tried it again. It sucked up all the ressources on my machines at a point I almost wanted to make a hard reboot. But it finally stopped and freed the memory as I have never seen on my machine, 1.5 GB used only on 16GB (see ULTIMATE TRY in pullAllPluginsSource.log). I'm though curious how Linux would react...

          Show
          jacques.le.roux Jacques Le Roux added a comment - I have quickly tried to create a pullAllPluginsSource task adding the code below Index: build.gradle =================================================================== --- build.gradle (revision 1783441) +++ build.gradle (working copy) @@ -745,6 +745,17 @@ } } + +task pullAllPluginsSource(group: ofbizPlugin, description: 'Download and install all plugins from source control') { + + String [] pluginsList = new File('pluginsList.txt') + for (plugin in pluginsList) { + def svnUrl = "https: //svn.apache.org/repos/asf/ofbiz/ofbiz-plugins/trunk/" + plugin + def workspaceDir = "${rootDir}/plugins/" + plugin + installPlugin(plugin) + } +} + // ========== Clean up tasks ========== task cleanCatalina(group: cleanupGroup, description: 'Clean Catalina data in runtime/catalina/work') { doLast { delete "${rootDir}/runtime/catalina/work" } But I got a weird issue. The task stacks the call to installPlugin and so also stacks Gradle Daemons (I use it locally). So the memory increases fast and at some point you either get ouf or memory, or if you care before the whole thing crashes anyway. See pullAllPluginsSource.log and pluginsList.txt attached files. It would have been quite cool but it's obviously not good. What would be the right way? BTW "don't try that at home" In a last attempt, I rebooted, cleared all possible memory and tried it again. It sucked up all the ressources on my machines at a point I almost wanted to make a hard reboot. But it finally stopped and freed the memory as I have never seen on my machine, 1.5 GB used only on 16GB (see ULTIMATE TRY in pullAllPluginsSource.log). I'm though curious how Linux would react...
          Hide
          taher Taher Alkhateeb added a comment -

          I'm already working on this task Jacques. There is no need to rush

          Show
          taher Taher Alkhateeb added a comment - I'm already working on this task Jacques. There is no need to rush
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Great, I quickly tried that to replace ofbiz-framework-buildbot that I use in Buildbot and demos, but that can wait indeed.

          Show
          jacques.le.roux Jacques Le Roux added a comment - Great, I quickly tried that to replace ofbiz-framework-buildbot that I use in Buildbot and demos, but that can wait indeed.
          Hide
          taher Taher Alkhateeb added a comment -

          Disabled the gradle daemon in the embedded plugins API in r1783852. This would help tackle any memory issues. Next, I am designing the pullAllPluginSource hopefully with an efficient algorithm in preparation for creating two buildbot scripts (one for framework and one for plugins).

          Show
          taher Taher Alkhateeb added a comment - Disabled the gradle daemon in the embedded plugins API in r1783852. This would help tackle any memory issues. Next, I am designing the pullAllPluginSource hopefully with an efficient algorithm in preparation for creating two buildbot scripts (one for framework and one for plugins).
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Disabled the gradle daemon in the embedded plugins API in r1783852.

          Good point Taher!

          Show
          jacques.le.roux Jacques Le Roux added a comment - Disabled the gradle daemon in the embedded plugins API in r1783852. Good point Taher!
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          BTW If ever you tried the same thing than me "at home" and get an issue with stopped but unremovable daemons, easy way: remove the daemon folder under your gradle folder (where the cache is also)

          Show
          jacques.le.roux Jacques Le Roux added a comment - BTW If ever you tried the same thing than me "at home" and get an issue with stopped but unremovable daemons, easy way: remove the daemon folder under your gradle folder (where the cache is also)
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          OK, not only that. It did not work for me. So I thought you needed to remove all your gradle folder (the one under your username, where the cache is also). Else, at least on Windows, Gradle is unusable. Any time you try to use it, it spawns as much as processes and consoles your machine can support just to the point it crashes. But unfortunately even removing gradle folder did not work. After(? )/While(? ) downloading the Internet, same issue again!

          From the daemon-xxxx.out.logs it seems the issue lies-in/is-related-with "build.gradle line: 1030", ie

          exec { commandLine gradleRunner, '--no-daemon', 'installPlugin', "-PpluginId=${pluginId}" }

          BWT, doing so I noticed this minor, unrelated point to the resources consumption issue I face, that you can also find in daemon-xxx.out.logq and is worth to be noted here than to forget.

          POM relocation to an other version number is not fully supported in Gradle : xml-apis:xml-apis:2.0.2 relocated to xml-apis:xml-apis:1.0.b2.
          Please update your dependency to directly use the correct version 'xml-apis:xml-apis:1.0.b2'.
          Resolution will only pick dependencies of the relocated element. Artifacts and other metadata will be ignored.

          Show
          jacques.le.roux Jacques Le Roux added a comment - OK, not only that. It did not work for me. So I thought you needed to remove all your gradle folder (the one under your username, where the cache is also). Else, at least on Windows, Gradle is unusable. Any time you try to use it, it spawns as much as processes and consoles your machine can support just to the point it crashes. But unfortunately even removing gradle folder did not work. After(? )/While(? ) downloading the Internet, same issue again! From the daemon-xxxx.out.logs it seems the issue lies-in/is-related-with "build.gradle line: 1030", ie exec { commandLine gradleRunner, '--no-daemon', 'installPlugin', "-PpluginId=${pluginId}" } BWT, doing so I noticed this minor, unrelated point to the resources consumption issue I face, that you can also find in daemon-xxx.out.logq and is worth to be noted here than to forget. POM relocation to an other version number is not fully supported in Gradle : xml-apis:xml-apis:2.0.2 relocated to xml-apis:xml-apis:1.0.b2. Please update your dependency to directly use the correct version 'xml-apis:xml-apis:1.0.b2'. Resolution will only pick dependencies of the relocated element. Artifacts and other metadata will be ignored.
          Hide
          jacques.le.roux Jacques Le Roux added a comment - - edited

          For now no other solutions than https://docs.gradle.org/current/userguide/gradle_daemon.html#sec:disabling_the_daemon

          Even when completely deleted, somehow the daemons keep a memory of my try to install plugins.

          Show
          jacques.le.roux Jacques Le Roux added a comment - - edited For now no other solutions than https://docs.gradle.org/current/userguide/gradle_daemon.html#sec:disabling_the_daemon Even when completely deleted, somehow the daemons keep a memory of my try to install plugins.
          Hide
          taher Taher Alkhateeb added a comment -

          You are still assuming a solution exists. I am not yet done with development on pullAllPluginSource. So either you are discussing something different or assuming I am done. If it is rhe first then it needs to be discussed elsewhere, if it is the second then you should wait for a proper implementation first

          Show
          taher Taher Alkhateeb added a comment - You are still assuming a solution exists. I am not yet done with development on pullAllPluginSource. So either you are discussing something different or assuming I am done. If it is rhe first then it needs to be discussed elsewhere, if it is the second then you should wait for a proper implementation first
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          No no, I'm not assuming anything. I don't need to use plugins. It was just the daemons which run amok and started again and again, continuning to try to install plugins despite I did only ask to run OFBiz. Daemons are optional, so it's not really an issue for me. Just that I was unlucky with my try and reading

          It is also possible to destabilize the Daemon (and build environment in general) by running builds that do not release resources correctly. This is a particularly poignant problem when using Microsoft Windows as it is less forgiving of programs that fail to close files after reading or writing.

          at https://docs.gradle.org/current/userguide/gradle_daemon.html I guess I got those issues you can have with daemons in Windows (I did not report it all, it was really exhausting).

          I only reported here for people who could cross the same in unrelated situations. I agree is not the best place for that, but since I started here...

          Show
          jacques.le.roux Jacques Le Roux added a comment - No no, I'm not assuming anything. I don't need to use plugins. It was just the daemons which run amok and started again and again, continuning to try to install plugins despite I did only ask to run OFBiz. Daemons are optional, so it's not really an issue for me. Just that I was unlucky with my try and reading It is also possible to destabilize the Daemon (and build environment in general) by running builds that do not release resources correctly. This is a particularly poignant problem when using Microsoft Windows as it is less forgiving of programs that fail to close files after reading or writing. at https://docs.gradle.org/current/userguide/gradle_daemon.html I guess I got those issues you can have with daemons in Windows (I did not report it all, it was really exhausting). I only reported here for people who could cross the same in unrelated situations. I agree is not the best place for that, but since I started here...
          Hide
          taher Taher Alkhateeb added a comment -

          Committed in r1786562 the implementation of the new task "pullAllPluginsSource" which delete the /plugins directory and replaces it with the plugins repository and for each plugin it executes the "install" task if it exists.

          The only thing left aside from general documentation is to create two separate buildbot scripts. One for ofbiz-framework and one for ofbiz-plugins to separate the two projects from each other.

          Show
          taher Taher Alkhateeb added a comment - Committed in r1786562 the implementation of the new task "pullAllPluginsSource" which delete the /plugins directory and replaces it with the plugins repository and for each plugin it executes the "install" task if it exists. The only thing left aside from general documentation is to create two separate buildbot scripts. One for ofbiz-framework and one for ofbiz-plugins to separate the two projects from each other.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Thanks Taher great work!

          I'll answer to your last related message in the dev ML. To not make my answer too verbose there, I have a question here.

          Maybe I read it wrong but reading the documentation I thought we would have a .svn directory in each plugin

          Download all officially supported plugins from source control (currently subversion) and place them inclusive of their ".svn" directory in /plugins.

          Actually we have only one . svn for all plugins, and I very like it like it like that, much easier for maintenance, thanks! So maybe we should make the documentation more clear?
          Also when I run pullAllPluginsSource I get

          C:\projectsASF\ofbiz-framework>gradlew pullAllPluginsSource
          :pullPluginsFromSvn
          :pullAllPluginsSource
          :installAllPlugins
          No install task defined for plugin assetmaint
          No install task defined for plugin bi
          No install task defined for plugin birt
          No install task defined for plugin cmssite
          No install task defined for plugin ebay
          No install task defined for plugin ecommerce
          No install task defined for plugin example
          No install task defined for plugin exampleext
          No install task defined for plugin hhfacility
          No install task defined for plugin ldap
          No install task defined for plugin lucene
          No install task defined for plugin myportal
          No install task defined for plugin passport
          No install task defined for plugin pricat
          No install task defined for plugin projectmgr
          No install task defined for plugin scrum
          No install task defined for plugin solr
          No install task defined for plugin webpos
          

          I guess it's OK (no specific tasks to do OOTB in each plugins) but maybe we should document that in the the README? To avoid questions on this subject (and prevent my RTFM reflex ) or make answers easier by pointing to the documentation

          Show
          jacques.le.roux Jacques Le Roux added a comment - Thanks Taher great work! I'll answer to your last related message in the dev ML. To not make my answer too verbose there, I have a question here. Maybe I read it wrong but reading the documentation I thought we would have a .svn directory in each plugin Download all officially supported plugins from source control (currently subversion) and place them inclusive of their ".svn" directory in /plugins. Actually we have only one . svn for all plugins, and I very like it like it like that, much easier for maintenance, thanks! So maybe we should make the documentation more clear? Also when I run pullAllPluginsSource I get C:\projectsASF\ofbiz-framework>gradlew pullAllPluginsSource :pullPluginsFromSvn :pullAllPluginsSource :installAllPlugins No install task defined for plugin assetmaint No install task defined for plugin bi No install task defined for plugin birt No install task defined for plugin cmssite No install task defined for plugin ebay No install task defined for plugin ecommerce No install task defined for plugin example No install task defined for plugin exampleext No install task defined for plugin hhfacility No install task defined for plugin ldap No install task defined for plugin lucene No install task defined for plugin myportal No install task defined for plugin passport No install task defined for plugin pricat No install task defined for plugin projectmgr No install task defined for plugin scrum No install task defined for plugin solr No install task defined for plugin webpos I guess it's OK (no specific tasks to do OOTB in each plugins) but maybe we should document that in the the README? To avoid questions on this subject (and prevent my RTFM reflex ) or make answers easier by pointing to the documentation
          Hide
          taher Taher Alkhateeb added a comment -

          To answer quickly:

          • if you use pullPluginSource, then each pulled plugin will have its own .svn. If on the other hand you use pullAllPluginsSource then you will have a single .svn for all plugins
          • The no install task defined for plugin <whatever> means there is nothing to do. To make it more clear, we can simply add the words "nothing to do" to make it clear that nothing went wrong. This is a minor issue as far as I'm concerned though.
          Show
          taher Taher Alkhateeb added a comment - To answer quickly: if you use pullPluginSource, then each pulled plugin will have its own .svn. If on the other hand you use pullAllPluginsSource then you will have a single .svn for all plugins The no install task defined for plugin <whatever> means there is nothing to do. To make it more clear, we can simply add the words "nothing to do" to make it clear that nothing went wrong. This is a minor issue as far as I'm concerned though.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          if you use pullPluginSource, then each pulled plugin will have its own .svn. If on the other hand you use pullAllPluginsSource then you will have a single .svn for all plugins

          I think we need to document this technical aspect in the README. I'll use your sentence.

          The no install task defined for plugin <whatever> means there is nothing to do. To make it more clear, we can simply add the words "nothing to do" to make it clear that nothing went wrong. This is a minor issue as far as I'm concerned though.

          Yes sounds good to me, I'd add ". Nothing to do and worry about, see the documentation for more"

          Show
          jacques.le.roux Jacques Le Roux added a comment - if you use pullPluginSource, then each pulled plugin will have its own .svn. If on the other hand you use pullAllPluginsSource then you will have a single .svn for all plugins I think we need to document this technical aspect in the README. I'll use your sentence. The no install task defined for plugin <whatever> means there is nothing to do. To make it more clear, we can simply add the words "nothing to do" to make it clear that nothing went wrong. This is a minor issue as far as I'm concerned though. Yes sounds good to me, I'd add ". Nothing to do and worry about, see the documentation for more"
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Done

          Show
          jacques.le.roux Jacques Le Roux added a comment - Done
          Hide
          taher Taher Alkhateeb added a comment -

          Work in here is more-or-less done. The plugin system is now fully in place, buildbot is updated, and the plugin API is complete.

          What remains should go in new JIRAs including:

          • splitting buildbot into two scripts, one for ofbiz-framework and one for ofbiz-plugins
          • deciding on release strategy
          • setting up a maven repo
          • removing the dependencies of ofbiz-framework on ofbiz-plugins

          WIP, will issue other JIRAs later

          Show
          taher Taher Alkhateeb added a comment - Work in here is more-or-less done. The plugin system is now fully in place, buildbot is updated, and the plugin API is complete. What remains should go in new JIRAs including: splitting buildbot into two scripts, one for ofbiz-framework and one for ofbiz-plugins deciding on release strategy setting up a maven repo removing the dependencies of ofbiz-framework on ofbiz-plugins WIP, will issue other JIRAs later

            People

            • Assignee:
              taher Taher Alkhateeb
              Reporter:
              taher Taher Alkhateeb
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development