Bigtop
  1. Bigtop
  2. BIGTOP-1263

Next vagrant iteration: Make easy to customize,maintain and a HA recipe.

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.7.0
    • Fix Version/s: backlog
    • Component/s: vm
    • Labels:
      None

      Description

      BIGTOP-1261 now demonstrates the need for HA vagrant recipe for development and testing.

      After we get BIGTOP-1178 in, Lets come up with a documentation on how to create custom vagrant recipes. Specifically we want:

      • HA HDFS cluster. option to the vagrant recipes.
      • vagrant based EC2 deployment
        others...

      Also in this JIRA we should generally review vagrant and think about the new features in the new Vagrant API (like the pull and push stuff).

        Activity

        Hide
        jay vyas added a comment -

        For now, we are just adding new recipes under the vagrant directory, and i guess thats okay.

        At some point this JIRA can clean them all up.

        1) I hope its okay with the other bigtop members that we are adding all these vagrant recipes ? Don't want to clutter it up too much !

        2) Should we use bash for these, or is something more elegant in order like maybe the gradle - vagrant plugin www.coherentsoftware.com.au/node/36 ?

        Show
        jay vyas added a comment - For now, we are just adding new recipes under the vagrant directory, and i guess thats okay. At some point this JIRA can clean them all up. 1) I hope its okay with the other bigtop members that we are adding all these vagrant recipes ? Don't want to clutter it up too much ! 2) Should we use bash for these, or is something more elegant in order like maybe the gradle - vagrant plugin www.coherentsoftware.com.au/node/36 ?
        Hide
        Evans Ye added a comment -

        jay vyas, I think the idea is great since there're too many combinations when it comes to deploy a hadoop cluster. Providing some classic deployment packs(like a HA cluster) not only makes people easily to understand how to use bigtop-puppet but also helps us to test out those deploy combinations when we're going to release a new version of hadoop and its ecosystem.
        Hi Yao-Tsung Wang, maybe you can just fire up another ticket which is specific for ec2 deployment implementation an link together with this one.

        Show
        Evans Ye added a comment - jay vyas , I think the idea is great since there're too many combinations when it comes to deploy a hadoop cluster. Providing some classic deployment packs(like a HA cluster) not only makes people easily to understand how to use bigtop-puppet but also helps us to test out those deploy combinations when we're going to release a new version of hadoop and its ecosystem. Hi Yao-Tsung Wang , maybe you can just fire up another ticket which is specific for ec2 deployment implementation an link together with this one.
        Hide
        Yao-Tsung Wang added a comment -

        Hi Jay, recently I just upload some code based on vagrant-aws to run bigtop on AWS EC2.
        The first one is based on Ubuntu.
        https://github.com/jazzwang/vagrant-hadoop/tree/master/bigtop-aws/ubuntu
        I'll try to write another one based on current vagrant-puppet code.

        Show
        Yao-Tsung Wang added a comment - Hi Jay, recently I just upload some code based on vagrant-aws to run bigtop on AWS EC2. The first one is based on Ubuntu. https://github.com/jazzwang/vagrant-hadoop/tree/master/bigtop-aws/ubuntu I'll try to write another one based on current vagrant-puppet code.
        Hide
        jay vyas added a comment -

        yup ! but once we add the EC2 provisioner, maybe we can do even more with them.

        Show
        jay vyas added a comment - yup ! but once we add the EC2 provisioner, maybe we can do even more with them.
        Hide
        Roman Shaposhnik added a comment -

        jay vyas let me see if I get this straight – you're saying that vagrant scripts are essentially for a sort of dev-centric quick and dirty instances of various Hadoop &co configurations, right?

        Show
        Roman Shaposhnik added a comment - jay vyas let me see if I get this straight – you're saying that vagrant scripts are essentially for a sort of dev-centric quick and dirty instances of various Hadoop &co configurations, right?
        Hide
        jay vyas added a comment - - edited
        • For HA, yes you are right.
        • But for some stuff, like EC2 provisioning, we need more than editing puppet confs.

        In any case: The main purpose here is to make it so that we can easily build bigtop clusters which are aimed at minimally representing a particaular feature or bug. For example, on monday, maybe i want to test HA. On tuesday, maybe im in HBase mode. .... So lets make these vagrant recipes more reusable, and easier to extend.

        Right now we just have one monolithic vagrant provisioner.

        ill try to sketch a more concrete idea of what im thinking, but for now, something like this:

        bigtop/ 
            bigtop-deploy/
                 vagrant/ 
                      vagrant-puppet/
                              base-provisioner.sh
                                vagrant-HA-2nod-vbox/
                                      custom-provisioner.sh
                                vagrant-EC2-10-node-cluster/
                                      custom-provisioner.sh
                                 BIGTOP-1221/
                                      custom-provisioner.sh
        

        So basicallly, to play with a particular hadoop deployment, you just create a new subdirectory, and make some minor customizations and evans and I will try to maintain the base-provisioner script.

        In the end some of these sub directories might be great to contribute back to bigtop, some might just be private.

        Show
        jay vyas added a comment - - edited For HA, yes you are right. But for some stuff, like EC2 provisioning, we need more than editing puppet confs. In any case: The main purpose here is to make it so that we can easily build bigtop clusters which are aimed at minimally representing a particaular feature or bug. For example, on monday, maybe i want to test HA. On tuesday, maybe im in HBase mode. .... So lets make these vagrant recipes more reusable, and easier to extend. Right now we just have one monolithic vagrant provisioner. ill try to sketch a more concrete idea of what im thinking, but for now, something like this: bigtop/ bigtop-deploy/ vagrant/ vagrant-puppet/ base-provisioner.sh vagrant-HA-2nod-vbox/ custom-provisioner.sh vagrant-EC2-10-node-cluster/ custom-provisioner.sh BIGTOP-1221/ custom-provisioner.sh So basicallly, to play with a particular hadoop deployment, you just create a new subdirectory, and make some minor customizations and evans and I will try to maintain the base-provisioner script. In the end some of these sub directories might be great to contribute back to bigtop, some might just be private.
        Hide
        Konstantin Boudnik added a comment -

        For vagrant configuration - can't you rely on the standard Bigtop puppet deployment? Just trying to wrap my head around it....

        Show
        Konstantin Boudnik added a comment - For vagrant configuration - can't you rely on the standard Bigtop puppet deployment? Just trying to wrap my head around it....
        Hide
        Konstantin Boudnik added a comment -

        Component and Affects version(s) fields please

        Show
        Konstantin Boudnik added a comment - Component and Affects version(s) fields please

          People

          • Assignee:
            Unassigned
            Reporter:
            jay vyas
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development