Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.1 (Alpha)
    • Component/s: Core system
    • Labels:
      None

      Description

      Instead of processing the builds sequentially it would be great to be
      able to specify how many projects are being build concurrently.

      1. continuum-265-docs.patch
        3 kB
        jzurbano
      2. CONTINUUM-265-UI.patch
        33 kB
        Gwen Harold Autencio
      3. continuum-265-webapp.patch
        5 kB
        jzurbano
      4. more-update-CONTINUUM-265-UI.patch
        32 kB
        Gwen Harold Autencio
      5. parallelBuilds-screenshots.zip
        11 kB
        jzurbano
      6. schedule-problem.patch
        5 kB
        Gwen Harold Autencio
      7. update-CONTINUUM-265-UI.patch
        31 kB
        Gwen Harold Autencio

        Issue Links

          Activity

          Hide
          Christian E Gruber added a comment -

          This would be highly useful, especially in situations where you have peers in a dependency chain, but would require a fair bit of dependency management.

          Consider:
          Project A
          BCD depend on A

          E depends on C and D
          F depends on B and C

          G depends on E and F

          A would have to be built in isolation, but B, C, and D could be built simultaneously. E and/or F could be built, only with BOTH of their respective dependencies are done. One way would be to fork threads for each build. Each would know their dependencies and poll them until they receive a "yes, I'm done", then they could begin on their merry way.

          A key problem with this, is that Maven assumes that all build artifacts go to the central repository, so relying on maven itself won't be a solution. Continuum would have to maintain more build state. This should be fine, since it already scoops up dependencies and displays them, but the point is that it would have to be a Continuum solution, not a maven solution. Also, if it were, and if dependencies for ant/shell build types could be added manually, as long as those dependencies were also in the current Continuum instance, then such a solution could work for any and all project types.

          Show
          Christian E Gruber added a comment - This would be highly useful, especially in situations where you have peers in a dependency chain, but would require a fair bit of dependency management. Consider: Project A BCD depend on A E depends on C and D F depends on B and C G depends on E and F A would have to be built in isolation, but B, C, and D could be built simultaneously. E and/or F could be built, only with BOTH of their respective dependencies are done. One way would be to fork threads for each build. Each would know their dependencies and poll them until they receive a "yes, I'm done", then they could begin on their merry way. A key problem with this, is that Maven assumes that all build artifacts go to the central repository, so relying on maven itself won't be a solution. Continuum would have to maintain more build state. This should be fine, since it already scoops up dependencies and displays them, but the point is that it would have to be a Continuum solution, not a maven solution. Also, if it were, and if dependencies for ant/shell build types could be added manually, as long as those dependencies were also in the current Continuum instance, then such a solution could work for any and all project types.
          Hide
          Yan Shapochnik added a comment -

          I believe this would extremely useful, even for shell script projects. For example, I have 4 groups of projects setup. Each group represents a platform and I need to build each group of projects concurrently, since each group is built on a separate platform. I am not using maven or ant. I am actually using continuum to build c++ code and it works quite well, except no parallel build ability appears to exist. It would be good to have the ability to define a separate build executor for each project group.

          Show
          Yan Shapochnik added a comment - I believe this would extremely useful, even for shell script projects. For example, I have 4 groups of projects setup. Each group represents a platform and I need to build each group of projects concurrently, since each group is built on a separate platform. I am not using maven or ant. I am actually using continuum to build c++ code and it works quite well, except no parallel build ability appears to exist. It would be good to have the ability to define a separate build executor for each project group.
          Hide
          oenseling added a comment -

          We are managing 100+ projects with a single instance of continuum. I desperately need the ability to scale our CI up to multiple build servers.
          Currently we are just building java code and running unit tests. Individual builds take a few minutes each on average. But now we want to add nightly builds with integration tests and these beast will likely take more than one hour each.

          Running multiple instances of continuum is not a nice option for us because we like the ability to manage all our CI builds centrally.

          Show
          oenseling added a comment - We are managing 100+ projects with a single instance of continuum. I desperately need the ability to scale our CI up to multiple build servers. Currently we are just building java code and running unit tests. Individual builds take a few minutes each on average. But now we want to add nightly builds with integration tests and these beast will likely take more than one hour each. Running multiple instances of continuum is not a nice option for us because we like the ability to manage all our CI builds centrally.
          Hide
          Baptiste MATHUS added a comment -

          The big problem that comes into mind in the repository locking. In fact, what will happen if two separated maven instances download/access/modify the same file concurrently from the local repository.
          As Wendy said, either we

          • need to handle more than one repository
          • wait for this related bug/improvement to be fixed

          I'm no maven expert, but I guess there's a risk to see a lot of new problems if the local repository is not unique. What about new maven instances? Which local repository are they going to use? And so on.

          Show
          Baptiste MATHUS added a comment - The big problem that comes into mind in the repository locking. In fact, what will happen if two separated maven instances download/access/modify the same file concurrently from the local repository. As Wendy said, either we need to handle more than one repository wait for this related bug/improvement to be fixed I'm no maven expert, but I guess there's a risk to see a lot of new problems if the local repository is not unique. What about new maven instances? Which local repository are they going to use? And so on.
          Hide
          deckrider added a comment -

          I urgently need this also.

          There should be configuration to permit N number of concurrent builds, and for each one defined, a different local repository must be specified, or at least a different settings.xml, which then could specify a different repository.

          This will avoid any problem with waiting until MNG-2802 is fixed.

          Show
          deckrider added a comment - I urgently need this also. There should be configuration to permit N number of concurrent builds, and for each one defined, a different local repository must be specified, or at least a different settings.xml, which then could specify a different repository. This will avoid any problem with waiting until MNG-2802 is fixed.
          Hide
          Christian E Gruber added a comment -

          Wait - how does Maven handle concurrent access to the local repo. I've often been building several projects in several windows simultaneously, so I wonder how maven handles the conflicts.

          Show
          Christian E Gruber added a comment - Wait - how does Maven handle concurrent access to the local repo. I've often been building several projects in several windows simultaneously, so I wonder how maven handles the conflicts.
          Hide
          Christian E Gruber added a comment -

          Never mind - I looked it up. However, some of this could be resolved by allowing for any projects that use different local repos to be potentially parallelized. Or asserting that each group HAS its own local repo unless overridden, and allowing different groups to be parallel.

          I know it's not that simple, because you need to manage thread pools and workers to do the builds and such.

          A bigger issue is that it requires a stronger build-order system than a forced linear ranking based on dependency. It would require a build tree where nodes could be pulled off and parallelized and then some logic for finding the synchronization points.

          Show
          Christian E Gruber added a comment - Never mind - I looked it up. However, some of this could be resolved by allowing for any projects that use different local repos to be potentially parallelized. Or asserting that each group HAS its own local repo unless overridden, and allowing different groups to be parallel. I know it's not that simple, because you need to manage thread pools and workers to do the builds and such. A bigger issue is that it requires a stronger build-order system than a forced linear ranking based on dependency. It would require a build tree where nodes could be pulled off and parallelized and then some logic for finding the synchronization points.
          Hide
          deckrider added a comment -

          I see in http://continuum.apache.org/docs/1.2/release-notes.html that "now it's possible to configure a local m2 repository per project group". This is excellent!

          Now as an interim measure, can each project group run builds in parallel with other project groups?

          Show
          deckrider added a comment - I see in http://continuum.apache.org/docs/1.2/release-notes.html that "now it's possible to configure a local m2 repository per project group". This is excellent! Now as an interim measure, can each project group run builds in parallel with other project groups?
          Hide
          Maria Catherine Tan added a comment -

          No, it's not yet possible.

          Show
          Maria Catherine Tan added a comment - No, it's not yet possible.
          Hide
          Emmanuel Venisse added a comment -

          The problem with concurrent builds is dependent projects.

          If project A depends on project B depends on project C, it isn't possible/recommended to parallelize builds. But if project A depends on B and C, it should be possible to parallelize B and C builds.

          Show
          Emmanuel Venisse added a comment - The problem with concurrent builds is dependent projects. If project A depends on project B depends on project C, it isn't possible/recommended to parallelize builds. But if project A depends on B and C, it should be possible to parallelize B and C builds.
          Hide
          aaron pieper added a comment -

          Vulcan is another open-source Java project for CI, it has a build manager which supports multiple threads, and manages complex dependency relationships like you're mentioning here. I don't know if any of the code would be useful for this project, or whether it can be reused. It's been released under the GPLv2.

          http://code.google.com/p/vulcan/

          Show
          aaron pieper added a comment - Vulcan is another open-source Java project for CI, it has a build manager which supports multiple threads, and manages complex dependency relationships like you're mentioning here. I don't know if any of the code would be useful for this project, or whether it can be reused. It's been released under the GPLv2. http://code.google.com/p/vulcan/
          Hide
          Torsten Curdt added a comment -

          How I would tackle it: Every checkout/project gets it's own local repo. Then there should be no problem at all with locking. One could even clean out those repos on every build (if desired) which might be great for making sure all dependencies are really available.

          Show
          Torsten Curdt added a comment - How I would tackle it: Every checkout/project gets it's own local repo. Then there should be no problem at all with locking. One could even clean out those repos on every build (if desired) which might be great for making sure all dependencies are really available.
          Hide
          Maria Odea Ching added a comment - - edited

          I've posted a draft proposal for this feature here:
          http://cwiki.apache.org/confluence/display/CONTINUUM/Draft+-+Continuum+Parallel+or+Concurrent+Builds

          Comments and suggestions are much welcome..
          Thanks!

          Show
          Maria Odea Ching added a comment - - edited I've posted a draft proposal for this feature here: http://cwiki.apache.org/confluence/display/CONTINUUM/Draft+-+Continuum+Parallel+or+Concurrent+Builds Comments and suggestions are much welcome.. Thanks!
          Hide
          Gwen Harold Autencio added a comment -

          Hi. Attached a partial patch for UI requirements.

          Show
          Gwen Harold Autencio added a comment - Hi. Attached a partial patch for UI requirements.
          Hide
          Gwen Harold Autencio added a comment -

          Hi.. I attach an update patch. updated UI/Configuration #1-4
          Not included. #5 queue page should list average wait time for each queue.
          Thanks.

          Show
          Gwen Harold Autencio added a comment - Hi.. I attach an update patch. updated UI/Configuration #1-4 Not included. #5 queue page should list average wait time for each queue. Thanks.
          Hide
          Maria Odea Ching added a comment - - edited

          Patch for UI (update-CONTINUUM-265-UI.patch) applied in continuum-parallel-builds branch -r726688. Thanks Gwen

          Show
          Maria Odea Ching added a comment - - edited Patch for UI (update- CONTINUUM-265 -UI.patch) applied in continuum-parallel-builds branch -r726688. Thanks Gwen
          Hide
          Gwen Harold Autencio added a comment -

          Hi... found a problem on my previous patch, the list of build queues in the edit schedule page can't be added in the Schedule. Attached patch fixed the problem.
          Thanks.

          Show
          Gwen Harold Autencio added a comment - Hi... found a problem on my previous patch, the list of build queues in the edit schedule page can't be added in the Schedule. Attached patch fixed the problem. Thanks.
          Hide
          Gwen Harold Autencio added a comment -

          attached fix for adding build queues in schedule.

          Show
          Gwen Harold Autencio added a comment - attached fix for adding build queues in schedule.
          Hide
          Maria Odea Ching added a comment -

          schedule-problem.patch applied to continuum-parallel-builds-branch -r726973. Thanks again Gwen!

          Show
          Maria Odea Ching added a comment - schedule-problem.patch applied to continuum-parallel-builds-branch -r726973. Thanks again Gwen!
          Hide
          jzurbano added a comment -

          Patch includes:

          • displays the missing checkboxes
          • added duplicate checking when adding build queues
          • modified checking of the number of build queues allowed
          • fixed the action error display
          Show
          jzurbano added a comment - Patch includes: displays the missing checkboxes added duplicate checking when adding build queues modified checking of the number of build queues allowed fixed the action error display
          Hide
          Maria Odea Ching added a comment -

          continuum-265-webapp.patch applied to continuum-parallel-builds branch in -r732596. Thanks Jevica!

          Show
          Maria Odea Ching added a comment - continuum-265-webapp.patch applied to continuum-parallel-builds branch in -r732596. Thanks Jevica!
          Hide
          jzurbano added a comment - - edited

          Attached documentation (continuum-265-docs.patch) and screen shots (parallelBuilds-screenshots.zip).

          Show
          jzurbano added a comment - - edited Attached documentation (continuum-265-docs.patch) and screen shots (parallelBuilds-screenshots.zip).
          Hide
          Maria Odea Ching added a comment -

          Hi All,

          I've staged another snapshot of the parallel builds branch (-r733009) at:

          http://people.apache.org/~oching/continuum-parallel-builds-r733009/org/apache/continuum/continuum-jetty/1.3.1-SNAPSHOT/

          To get it working:
          1. In the Configuration page, set Number of Allowed Builds in Parallel to a value greater than 1 and save the changes.
          2. Create a new Build Queue. You can create a few more if you want but you shouldn't be permitted to create more than the Number of Allowed Builds in Parallel you've set in the configuration.
          3. In the Schedules page, edit DEFAULT_SCHEDULE and select the Build Queues you want to attach to it. These build queues would be used to distribute the checkouts/builds to be executed when a build is triggered (either forced or scheduled as long as the build definition used has the DEFAULT_SCHEDULE attached to it).
          4. Add a number of projects (different project groups if possible) to Continuum. You should be able to see in the Queues page the current checkout/builds executing and the queued tasks, and to which Build Queue are these tasks executing or queued.

          Step 3 is actually optional. If there are no build queues attached to a schedule, the parallel builds manager would use the DEFAULT_BUILD_QUEUE. But in order to demonstrate concurrent builds, we must attach multiple build queues to the schedule.

          Comments, suggestions, feedback are welcome

          If there are no major bugs found or no further changes in the implementation, I say we are ready to merge the branch to trunk..

          Show
          Maria Odea Ching added a comment - Hi All, I've staged another snapshot of the parallel builds branch (-r733009) at: http://people.apache.org/~oching/continuum-parallel-builds-r733009/org/apache/continuum/continuum-jetty/1.3.1-SNAPSHOT/ To get it working: 1. In the Configuration page, set Number of Allowed Builds in Parallel to a value greater than 1 and save the changes. 2. Create a new Build Queue. You can create a few more if you want but you shouldn't be permitted to create more than the Number of Allowed Builds in Parallel you've set in the configuration. 3. In the Schedules page, edit DEFAULT_SCHEDULE and select the Build Queues you want to attach to it. These build queues would be used to distribute the checkouts/builds to be executed when a build is triggered (either forced or scheduled as long as the build definition used has the DEFAULT_SCHEDULE attached to it). 4. Add a number of projects (different project groups if possible) to Continuum. You should be able to see in the Queues page the current checkout/builds executing and the queued tasks, and to which Build Queue are these tasks executing or queued. Step 3 is actually optional. If there are no build queues attached to a schedule, the parallel builds manager would use the DEFAULT_BUILD_QUEUE. But in order to demonstrate concurrent builds, we must attach multiple build queues to the schedule. Comments, suggestions, feedback are welcome If there are no major bugs found or no further changes in the implementation, I say we are ready to merge the branch to trunk..
          Hide
          deckrider added a comment -

          Thank you. I am trying this. But I'm confused with respect to this:

          """
          3. In the Schedules page, edit DEFAULT_SCHEDULE and select the Build Queues you want to attach to it. These build queues would be used to distribute the checkouts/builds to be executed when a build is triggered (either forced or scheduled as long as the build definition used has the DEFAULT_SCHEDULE attached to it).

          Step 3 is actually optional. If there are no build queues attached to a schedule, the parallel builds manager would use the DEFAULT_BUILD_QUEUE. But in order to demonstrate concurrent builds, we must attach multiple build queues to the schedule.
          """

          I see my default queue and additional queue when I edit default schedule. But I don't see how to select both to add. I've tried shift click, etc. and then save, but the log does not appear to show that I've selected them.

          Also how is this optional if one wants to have parallel builds?

          Finally, how does this interact with the local repository? Is it expected that we must set up a repository for each queue to avoid potential corruption? If so, why not make the UI reflect this and simplify setup?

          Show
          deckrider added a comment - Thank you. I am trying this. But I'm confused with respect to this: """ 3. In the Schedules page, edit DEFAULT_SCHEDULE and select the Build Queues you want to attach to it. These build queues would be used to distribute the checkouts/builds to be executed when a build is triggered (either forced or scheduled as long as the build definition used has the DEFAULT_SCHEDULE attached to it). Step 3 is actually optional. If there are no build queues attached to a schedule, the parallel builds manager would use the DEFAULT_BUILD_QUEUE. But in order to demonstrate concurrent builds, we must attach multiple build queues to the schedule. """ I see my default queue and additional queue when I edit default schedule. But I don't see how to select both to add. I've tried shift click, etc. and then save, but the log does not appear to show that I've selected them. Also how is this optional if one wants to have parallel builds? Finally, how does this interact with the local repository? Is it expected that we must set up a repository for each queue to avoid potential corruption? If so, why not make the UI reflect this and simplify setup?
          Hide
          Wendy Smoak added a comment -

          I had trouble getting my selections to "take" in the multi-select box to associate queues with a schedule. I could click and shift-down-arrow to turn them blue (selected) and save, but when I returned, only the default one was selected. I repeated it, (possibly using cmd-click this time?) and then it seemed to save.

          I added several projects and saw the checkouts go into the three different queues, though I can't say I ever saw it checking out more than one project at a time. Most of my checkouts went into the default queue though, due to the problem mentioned above.

          The Queues page still does not always reflect the actual state of things. I forced builds on everything ... the log shows it happily updating and building away, but the Queues page does not list any projects. (I have a screen shot if you want it...) Some time later, it looks more reasonable, with three projects building, and lots of them listed below waiting their turn.

          I think the underlying functionality is working, but the UI isn't quite right.

          Show
          Wendy Smoak added a comment - I had trouble getting my selections to "take" in the multi-select box to associate queues with a schedule. I could click and shift-down-arrow to turn them blue (selected) and save, but when I returned, only the default one was selected. I repeated it, (possibly using cmd-click this time?) and then it seemed to save. I added several projects and saw the checkouts go into the three different queues, though I can't say I ever saw it checking out more than one project at a time. Most of my checkouts went into the default queue though, due to the problem mentioned above. The Queues page still does not always reflect the actual state of things. I forced builds on everything ... the log shows it happily updating and building away, but the Queues page does not list any projects. (I have a screen shot if you want it...) Some time later, it looks more reasonable, with three projects building, and lots of them listed below waiting their turn. I think the underlying functionality is working, but the UI isn't quite right.
          Hide
          Maria Odea Ching added a comment -

          Hi deckrider: by #3 being optional I meant that if you don't attach any build queues to the schedule the build would technically not be parallel and only the default build queue would be used for building the projects. As for the local repository question, you don't need to setup separate local repositories for each build queue as building of inter-dependent projects is not yet supported (this is for future enhancement) and there's also the limitation of 'A project group cannot be built multiple times simultaneously' as specified in the requirements doc: http://cwiki.apache.org/confluence/display/CONTINUUM/Continuum+Parallel+or+Concurrent+Builds

          Wendy, deckrider: Shift + click for selecting build queues seem to work for me.. hmm, could be a browser problem? May I ask what browser you're using? I'm using Firefox 2.0.0.17. I'll also try it in IE to see if I can replicate you problem..

          For the queues page, I think the update from scm is executed during the prepare build phase by the prepare build task executor (which IIRC was added for the transient state feature) so the project(s) is/are not in any of the checkout queues. Only after the update from scm does the project(s) get added to the build queue so that's the time when it would show up in the Queues page. On a related note, I think you'll be able to see the checkouts in the Queues page when you initially add projects to Continuum..

          Show
          Maria Odea Ching added a comment - Hi deckrider: by #3 being optional I meant that if you don't attach any build queues to the schedule the build would technically not be parallel and only the default build queue would be used for building the projects. As for the local repository question, you don't need to setup separate local repositories for each build queue as building of inter-dependent projects is not yet supported (this is for future enhancement) and there's also the limitation of 'A project group cannot be built multiple times simultaneously' as specified in the requirements doc: http://cwiki.apache.org/confluence/display/CONTINUUM/Continuum+Parallel+or+Concurrent+Builds Wendy, deckrider: Shift + click for selecting build queues seem to work for me.. hmm, could be a browser problem? May I ask what browser you're using? I'm using Firefox 2.0.0.17. I'll also try it in IE to see if I can replicate you problem.. For the queues page, I think the update from scm is executed during the prepare build phase by the prepare build task executor (which IIRC was added for the transient state feature) so the project(s) is/are not in any of the checkout queues. Only after the update from scm does the project(s) get added to the build queue so that's the time when it would show up in the Queues page. On a related note, I think you'll be able to see the checkouts in the Queues page when you initially add projects to Continuum..
          Hide
          Maria Odea Ching added a comment -

          I tried selecting build queues in IE and Safari then Save, and both seems to work fine as well. However, when I selected all in the list box including the "-Build Queues-" list heading, I couldn't save the changes. Could this be what happened in your case deckrider? This also occurs in Firefox, btw. I'll just remove the list heading to avoid this..

          Show
          Maria Odea Ching added a comment - I tried selecting build queues in IE and Safari then Save, and both seems to work fine as well. However, when I selected all in the list box including the "- Build Queues -" list heading, I couldn't save the changes. Could this be what happened in your case deckrider? This also occurs in Firefox, btw. I'll just remove the list heading to avoid this..
          Hide
          Wendy Smoak added a comment -

          The multi-select list seems like it's going to be very easy to accidentally change as you're tabbing through the page. Would a different UI element work better here?

          It would also be good to add a column on the Schedules page and show the list of queues associated with that schedule.

          Show
          Wendy Smoak added a comment - The multi-select list seems like it's going to be very easy to accidentally change as you're tabbing through the page. Would a different UI element work better here? It would also be good to add a column on the Schedules page and show the list of queues associated with that schedule.
          Hide
          Maria Odea Ching added a comment -

          Yeah, maybe an option transfer list box is better suited instead of just a list box. +1 on adding a column on the Schedules page too..

          Show
          Maria Odea Ching added a comment - Yeah, maybe an option transfer list box is better suited instead of just a list box. +1 on adding a column on the Schedules page too..
          Hide
          Maria Odea Ching added a comment -

          Parallel builds branch merged to trunk in -r734099.

          Show
          Maria Odea Ching added a comment - Parallel builds branch merged to trunk in -r734099.

            People

            • Assignee:
              Maria Odea Ching
              Reporter:
              Torsten Curdt
            • Votes:
              22 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development