Bigtop
  1. Bigtop
  2. BIGTOP-710

Create a higher level orchestration deployment framework

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.4.0
    • Fix Version/s: None
    • Component/s: deployment
    • Labels:

      Description

      It would be very nice to start using our puppet code as a basis for a higher level orchestration framework capable of driving a true distributed multi-node deployment of Bigtop components. Puppet code is great at stamping out a predictable single node deployments, but we have to extend it with a DSL aimed at solving higher level orchestration.

      This JIRA is meant to be a central information radiator for collecting ideas on how to approach this (what DSL to use, how to structure communication between agents and fact sharing, etc).

        Issue Links

          Activity

          Hide
          Johnny Zhang added a comment -

          @Roman, does this related to package tests in anyway ?

          Show
          Johnny Zhang added a comment - @Roman, does this related to package tests in anyway ?
          Hide
          Bruno Mahé added a comment -

          Has anyone taken a look at mcollective? I haven't had time to try yet but from its front page http://puppetlabs.com/mcollective/introduction/ it seems to do exactly the intent of this ticket:

          The Marionette Collective AKA MCollective is a framework to build server orchestration or parallel job execution systems.

          It seems also to not be always tied to puppet:

          Break free from ever more complex naming conventions for hostnames as a means of identity. Use a very rich set of meta data provided by each machine to address them. Meta data comes from Puppet, Chef, Facter, Ohai or plugins you provide yourself.

          Show
          Bruno Mahé added a comment - Has anyone taken a look at mcollective? I haven't had time to try yet but from its front page http://puppetlabs.com/mcollective/introduction/ it seems to do exactly the intent of this ticket: The Marionette Collective AKA MCollective is a framework to build server orchestration or parallel job execution systems. It seems also to not be always tied to puppet: Break free from ever more complex naming conventions for hostnames as a means of identity. Use a very rich set of meta data provided by each machine to address them. Meta data comes from Puppet, Chef, Facter, Ohai or plugins you provide yourself.
          Hide
          Konstantin Boudnik added a comment - - edited

          I almost feel Glu
          And MCollectives too

          Show
          Konstantin Boudnik added a comment - - edited I almost feel Glu And MCollectives too
          Hide
          Konstantin Boudnik added a comment - - edited

          @Johnny: I don't think this is about system tests, although a part of any good orchestration framework would be a set of behavioral components tests that can provide a certain guarantees that brought up system/cluster is up to the specs, and working.

          Show
          Konstantin Boudnik added a comment - - edited @Johnny: I don't think this is about system tests, although a part of any good orchestration framework would be a set of behavioral components tests that can provide a certain guarantees that brought up system/cluster is up to the specs, and working.
          Hide
          Anatoli Fomenko added a comment -

          I'm wondering if the frameworks mentioned here would be suitable as a foundation for building a robust open cluster manager?

          Show
          Anatoli Fomenko added a comment - I'm wondering if the frameworks mentioned here would be suitable as a foundation for building a robust open cluster manager?
          Hide
          Bruno Mahé added a comment -

          Anatoli> It could, but I would not go there.
          Building a robust open cluster manager has a much larger scope than building something solving Apache Bigtop issues of today. Before trying to solve everyone's problem, I would rather try to solve Apache Bigtop's first. We can always expand later on.

          Which brings me to the point that this ticket try to state a solution to a problem not yet defined.
          We all know and talked about wanting some sort of orchestration layer to help us managing our VMs and test nodes (see BIGTOP-635 for instance) but we probably all have something different in mind. This may create some confusion later when implementations starts to appears but not matching everyone's expectations.
          So maybe we should spend some time first defining the use cases and requirements?

          Show
          Bruno Mahé added a comment - Anatoli> It could, but I would not go there. Building a robust open cluster manager has a much larger scope than building something solving Apache Bigtop issues of today. Before trying to solve everyone's problem, I would rather try to solve Apache Bigtop's first. We can always expand later on. Which brings me to the point that this ticket try to state a solution to a problem not yet defined. We all know and talked about wanting some sort of orchestration layer to help us managing our VMs and test nodes (see BIGTOP-635 for instance) but we probably all have something different in mind. This may create some confusion later when implementations starts to appears but not matching everyone's expectations. So maybe we should spend some time first defining the use cases and requirements?
          Hide
          Konstantin Boudnik added a comment -

          Bruno, good points. We certainly need to have a clear understanding of the goal, before we embark on any journey.

          I think this ticket is somewhat different. To my, the orchestration is about 'management of a bunch of nodes (i.e. cluster) via deployment, control and monitoring of BigTop produced artifacts'

          This is quite different from BIGTOP-635 and alike, IMO.

          Show
          Konstantin Boudnik added a comment - Bruno, good points. We certainly need to have a clear understanding of the goal, before we embark on any journey. I think this ticket is somewhat different. To my, the orchestration is about 'management of a bunch of nodes (i.e. cluster) via deployment, control and monitoring of BigTop produced artifacts' This is quite different from BIGTOP-635 and alike, IMO.
          Hide
          Bruno Mahé added a comment -

          Cos> I am not sure to follow. Where was it stated that this ticket is not somewhat different from BIGTOP-635? I stated they were related and that discussions about BIGTOP-635 brought up other discussions about such framework, but nothing further than that. My point being this ticket could be used as one of the implementation available for BIGTOP-635 and there may be a need for an api for the uses cases of this ticket.

          Show
          Bruno Mahé added a comment - Cos> I am not sure to follow. Where was it stated that this ticket is not somewhat different from BIGTOP-635 ? I stated they were related and that discussions about BIGTOP-635 brought up other discussions about such framework, but nothing further than that. My point being this ticket could be used as one of the implementation available for BIGTOP-635 and there may be a need for an api for the uses cases of this ticket.
          Hide
          Konstantin Boudnik added a comment -

          it has not been. I simply mentioned it to re-enforce my view of the orchestration and this it is different from what BIGTOP-635 is about. I agree that BIGTOP-710 would be a solid foundation of the other one. Sorry for any confusion

          Show
          Konstantin Boudnik added a comment - it has not been. I simply mentioned it to re-enforce my view of the orchestration and this it is different from what BIGTOP-635 is about. I agree that BIGTOP-710 would be a solid foundation of the other one. Sorry for any confusion

            People

            • Assignee:
              Roman Shaposhnik
              Reporter:
              Roman Shaposhnik
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:

                Development