Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: Deployment
    • Labels:

      Description

      The next iteration of the 1072 (vagrant recipe for deployment) would benefit alot from directly provisioning via puppet, rather than hardcoding the shell commands.

      HOW THE CURRENT PROVISiONER WORKS:

      The shell provisioner manually wget's the repos and installs components using yum.

      HOW THE PUPPET BASED PROVISIONER WILL WORK:

      In this JIRA, the aim will be to write a new provisioner that is puppet based, which simply manages a site.csv file for a default vagrant deployment. Then, the end users can update the site.csv, run "vagrant up", and have a puppet based distro up and running. The biggest advantage will be that the the vagrant deployer will continually be updated by the evolution of the puppet repos, rather than needing manual updates.

      1. BIGTOP-1171.2.patch
        4 kB
        Evans Ye
      2. BIGTOP-1171.1.patch
        4 kB
        Evans Ye

        Activity

        Hide
        Evans Ye added a comment -

        That will be so great! Please do so and give me some advice on it

        Show
        Evans Ye added a comment - That will be so great! Please do so and give me some advice on it
        Hide
        Sean Mackrory added a comment -

        Okay that makes sense. I'll see if I can get some time to look test / commit BIGTOP-1174 today or tomorrow.

        Show
        Sean Mackrory added a comment - Okay that makes sense. I'll see if I can get some time to look test / commit BIGTOP-1174 today or tomorrow.
        Hide
        Evans Ye added a comment - - edited

        Sean Mackrory
        Actually, just do vagrant up can get everything done with BIGTOP-1174 fixed. The reason you need to do vagrant provision --provision-with puppet again is that the current puppet snipes do not specify the executing order among hdfs daemons and init-hdfs.sh, so init-hdfs.sh failed to setup those hdfs directories when it executed before namenode and datanode are up and ready.
        The consequence is as what you saw, hbase-master can not detect its home folder on hdfs and hence abort itself.
        However, the second time of puppet deploy works because namenode and datanodes are all ready to go

        Show
        Evans Ye added a comment - - edited Sean Mackrory Actually, just do vagrant up can get everything done with BIGTOP-1174 fixed. The reason you need to do vagrant provision --provision-with puppet again is that the current puppet snipes do not specify the executing order among hdfs daemons and init-hdfs.sh , so init-hdfs.sh failed to setup those hdfs directories when it executed before namenode and datanode are up and ready. The consequence is as what you saw, hbase-master can not detect its home folder on hdfs and hence abort itself. However, the second time of puppet deploy works because namenode and datanodes are all ready to go
        Hide
        Sean Mackrory added a comment -

        I see. Let's add the `vagrant provision --provision-with puppet` step to the README then. I assumed provisioning was complete because the packages were installed and HDFS was initialized and everything...

        Show
        Sean Mackrory added a comment - I see. Let's add the `vagrant provision --provision-with puppet` step to the README then. I assumed provisioning was complete because the packages were installed and HDFS was initialized and everything...
        Hide
        Sean Mackrory added a comment -

        Committed. Nice contribution - I'm very excited to see this as a multi-node deployment!

        Show
        Sean Mackrory added a comment - Committed. Nice contribution - I'm very excited to see this as a multi-node deployment!
        Hide
        jay vyas added a comment -

        Sean Mackrory

        iirc should work perfectly:

        vagrant up ; #wait for it to finish
        vagrant provision --provision-with puppet
        
        Show
        jay vyas added a comment - Sean Mackrory iirc should work perfectly : vagrant up ; #wait for it to finish vagrant provision --provision-with puppet
        Hide
        Sean Mackrory added a comment -

        +1. Will commit shortly... It's all new stuff so it doesn't need to be perfect - it can't break any existing functionality. Some feedback, though, since you've already got plans to add to and improve this.

        • It didn't work because the HBase Master wasn't running. I had to run `sudo /usr/lib/hadoop/libexec/init-hdfs.sh` and `sudo service hbase-master restart` before the test succeeded.
        • Adding more than just Hadoop + HBase would be awesome, obviously, but that's definitely something that can be added later when someone has time to try out all the stuff that can be easily made to work out of the box this way.
        • I would suggest adding 'vagrant ssh' to the instructions and some examples of things that should work out of the box, rather than a script to test HBase.
        Show
        Sean Mackrory added a comment - +1. Will commit shortly... It's all new stuff so it doesn't need to be perfect - it can't break any existing functionality. Some feedback, though, since you've already got plans to add to and improve this. It didn't work because the HBase Master wasn't running. I had to run `sudo /usr/lib/hadoop/libexec/init-hdfs.sh` and `sudo service hbase-master restart` before the test succeeded. Adding more than just Hadoop + HBase would be awesome, obviously, but that's definitely something that can be added later when someone has time to try out all the stuff that can be easily made to work out of the box this way. I would suggest adding 'vagrant ssh' to the instructions and some examples of things that should work out of the box, rather than a script to test HBase.
        Hide
        jay vyas added a comment - - edited

        Thanks shawn...

        • We've (evans and I) have definetly tested this, but definetly welcome to have more testers
        • The only advantage of the old one is that its easy to understand for people that dont fully get puppet yet.. So Ive created a JIRA to delete the old provisioner and add comments to this one so that its easy to understand for puppet newcomers...

        (and i guess to evans above comment – as part of BIGTOP-1186 - we should add some instructions on how one might use puppet to provision simple as well as complex clusters)

        Show
        jay vyas added a comment - - edited Thanks shawn... We've (evans and I) have definetly tested this, but definetly welcome to have more testers The only advantage of the old one is that its easy to understand for people that dont fully get puppet yet.. So Ive created a JIRA to delete the old provisioner and add comments to this one so that its easy to understand for puppet newcomers... If you folks agree, we can push this in now and remove the old one in this new JIRA: https://issues.apache.org/jira/browse/BIGTOP-1186 . (and i guess to evans above comment – as part of BIGTOP-1186 - we should add some instructions on how one might use puppet to provision simple as well as complex clusters)
        Hide
        Evans Ye added a comment -

        Thanks Sean!
        Taking bigtop-1178 in to consider, I think this series of development mainly focus on building up a muti-nodes testing cluster, while origin one could be focused on spinning up a one node pseudo cluster. Both of them could be useful in different scenario, IMHO.

        Show
        Evans Ye added a comment - Thanks Sean! Taking bigtop-1178 in to consider, I think this series of development mainly focus on building up a muti-nodes testing cluster, while origin one could be focused on spinning up a one node pseudo cluster. Both of them could be useful in different scenario, IMHO.
        Hide
        Sean Mackrory added a comment -

        I'm testing it out right now. Should this replace the previous vagrant deployer? It seems like this would be superior, rather than just an alternative.

        Show
        Sean Mackrory added a comment - I'm testing it out right now. Should this replace the previous vagrant deployer? It seems like this would be superior, rather than just an alternative.
        Hide
        Evans Ye added a comment -

        Hello! Can some one have a look on this? Any comments will be appreciated

        Show
        Evans Ye added a comment - Hello! Can some one have a look on this? Any comments will be appreciated
        Hide
        jay vyas added a comment -
        Show
        jay vyas added a comment - bump https://issues.apache.org/jira/browse/BIGTOP-1178 awaits this one !
        Hide
        Peter Linnell added a comment -

        Roman, I'm back and will push it tonight, if I do not get bogged down. Need to keep my git-foo

        Show
        Peter Linnell added a comment - Roman, I'm back and will push it tonight, if I do not get bogged down. Need to keep my git-foo
        Hide
        Roman Shaposhnik added a comment -

        And just in case +1 from my side

        Show
        Roman Shaposhnik added a comment - And just in case +1 from my side
        Hide
        Roman Shaposhnik added a comment -

        Peter Linnell Peter, ping! Any chance you can help commit this?

        Show
        Roman Shaposhnik added a comment - Peter Linnell Peter, ping! Any chance you can help commit this?
        Hide
        Peter Linnell added a comment -

        I'll test this in the next 24 hours and if it works, I'll commit. Thanks for your efforts!

        Show
        Peter Linnell added a comment - I'll test this in the next 24 hours and if it works, I'll commit. Thanks for your efforts!
        Hide
        jay vyas added a comment -

        It's looking good and tested....
        Can we commit this? ...

        It's definitely ready I and Evans have both tested it: and it will significantly increase the
        Usability of these recipes to inherit the logic already in puppet.

        Also it will pave the way for BIGTOP-1178 (deploying puppetized bigtop clusters)!

        Show
        jay vyas added a comment - It's looking good and tested.... Can we commit this? ... It's definitely ready I and Evans have both tested it: and it will significantly increase the Usability of these recipes to inherit the logic already in puppet. Also it will pave the way for BIGTOP-1178 (deploying puppetized bigtop clusters)!
        Hide
        jay vyas added a comment -

        Okay, just tested and works .... results attached.
        Lets commit this so we can do the next iteration on this.
        Great work !

        jays-MacBook-Pro:vagrant-puppet jay$ ./hbase-test.sh 
        14/01/08 13:47:30 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
        HBase Shell; enter 'help<RETURN>' for list of supported commands.
        Type "exit<RETURN>" to leave the HBase Shell
        Version 0.94.12, rUnknown, Thu Oct 31 04:44:54 EDT 2013
        
        create 't1','cf1'
        0 row(s) in 2.7520 seconds
        
        put 't1', 'row1', 'cf1:q1', 'value1'
        0 row(s) in 0.2330 seconds
        
        scan 't1'
        ROW  COLUMN+CELL
         row1 column=cf1:q1, timestamp=1389188855489, value=value1
        1 row(s) in 0.1100 seconds
        
        disable 't1'
        0 row(s) in 1.1520 seconds
        
        drop 't1'
        0 row(s) in 1.0740 seconds
        
        Show
        jay vyas added a comment - Okay, just tested and works .... results attached. Lets commit this so we can do the next iteration on this. Great work ! jays-MacBook-Pro:vagrant-puppet jay$ ./hbase-test.sh 14/01/08 13:47:30 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available HBase Shell; enter 'help<RETURN>' for list of supported commands. Type "exit<RETURN>" to leave the HBase Shell Version 0.94.12, rUnknown, Thu Oct 31 04:44:54 EDT 2013 create 't1','cf1' 0 row(s) in 2.7520 seconds put 't1', 'row1', 'cf1:q1', 'value1' 0 row(s) in 0.2330 seconds scan 't1' ROW COLUMN+CELL row1 column=cf1:q1, timestamp=1389188855489, value=value1 1 row(s) in 0.1100 seconds disable 't1' 0 row(s) in 1.1520 seconds drop 't1' 0 row(s) in 1.0740 seconds
        Hide
        Evans Ye added a comment - - edited

        Hi Jay, thanks a million for the test & feedback.
        About the test procedure, sorry I didn't expain it clearly.
        Originally when you do vagrant up, it will automatically do vagrant provsion for you.
        The reason why you need to do vagrant provision --provision-with puppet again is that in our puppet snipes, init-hdfs script will be executed before hdfs daemons are up.
        When the second time you run puppet provisioner, hdfs daemons are all set(started by first try) so that the init-hdfs script can setup hdfs folders successfully and hbase master can be started up to work.
        This will be fixed by specifying the dependency on init-hdfs script which I've post in BIGTOP-1174.

        For the hbase-test script you just mentioned on your last comment, I'm thinking about why don't we use the same syntax as the test script you posted? So here comes patch version 2. This will be faster and more clearly than previous one.
        Here's the test output I got:

        $ ./hbase-test.sh
        14/01/08 03:09:39 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
        HBase Shell; enter 'help<RETURN>' for list of supported commands.
        Type "exit<RETURN>" to leave the HBase Shell
        Version 0.94.12, rUnknown, Thu Oct 31 04:44:54 EDT 2013
        
        create 't1','cf1'
        0 row(s) in 2.9400 seconds
        
        put 't1', 'row1', 'cf1:q1', 'value1'
        0 row(s) in 0.2470 seconds
        
        scan 't1'
        ROW  COLUMN+CELL
         row1 column=cf1:q1, timestamp=1389150584662, value=value1
        1 row(s) in 0.1230 seconds
        
        disable 't1'
        0 row(s) in 1.1350 seconds
        
        drop 't1'
        0 row(s) in 1.0710 seconds
        

        When something goes wrong, the output will show you what just the same as you're using hbase shell.
        How do you think, jay vyas?

        Show
        Evans Ye added a comment - - edited Hi Jay, thanks a million for the test & feedback. About the test procedure, sorry I didn't expain it clearly. Originally when you do vagrant up , it will automatically do vagrant provsion for you. The reason why you need to do vagrant provision --provision-with puppet again is that in our puppet snipes, init-hdfs script will be executed before hdfs daemons are up. When the second time you run puppet provisioner, hdfs daemons are all set(started by first try) so that the init-hdfs script can setup hdfs folders successfully and hbase master can be started up to work. This will be fixed by specifying the dependency on init-hdfs script which I've post in BIGTOP-1174 . For the hbase-test script you just mentioned on your last comment, I'm thinking about why don't we use the same syntax as the test script you posted? So here comes patch version 2. This will be faster and more clearly than previous one. Here's the test output I got: $ ./hbase-test.sh 14/01/08 03:09:39 WARN conf.Configuration: hadoop. native .lib is deprecated. Instead, use io. native .lib.available HBase Shell; enter 'help<RETURN>' for list of supported commands. Type "exit<RETURN>" to leave the HBase Shell Version 0.94.12, rUnknown, Thu Oct 31 04:44:54 EDT 2013 create 't1','cf1' 0 row(s) in 2.9400 seconds put 't1', 'row1', 'cf1:q1', 'value1' 0 row(s) in 0.2470 seconds scan 't1' ROW COLUMN+CELL row1 column=cf1:q1, timestamp=1389150584662, value=value1 1 row(s) in 0.1230 seconds disable 't1' 0 row(s) in 1.1350 seconds drop 't1' 0 row(s) in 1.0710 seconds When something goes wrong, the output will show you what just the same as you're using hbase shell. How do you think, jay vyas ?
        Hide
        jay vyas added a comment -

        Evans Ye

        hi evans. okay, ive finished testing , and i noticed that "vagrant up --provision-with puppet" fails, however

        vagrant up ; #wait for it to finish
         vagrant provision --provision-with puppet
        

        succeeds ! And it works great... Just for the commiters : here's a paste of my hbase results confirming that this patch is ready to roll .

        [root@vagrant vagrant]hbase shell <<EOF
        create 't2','f2' 
        put 't2', 'row1', 'f2:a', 'val1'
        scan 't2'
        EOF
        14/01/07 22:36:54 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
        create 't2','f2' 
        0 row(s) in 2.8070 seconds
        put 't2', 'row1', 'f2:a', 'val1'
        0 row(s) in 0.1010 seconds
        scan 't2'
        ROW                                  COLUMN+CELL                                                                                               
         row1                                column=f2:a, timestamp=1389134219249, value=val1                                                          
        1 row(s) in 0.0480 seconds
        

        Can we now commit this ? I think its ready.

        Show
        jay vyas added a comment - Evans Ye hi evans. okay, ive finished testing , and i noticed that "vagrant up --provision-with puppet" fails, however vagrant up ; #wait for it to finish vagrant provision --provision-with puppet succeeds ! And it works great... Just for the commiters : here's a paste of my hbase results confirming that this patch is ready to roll . [root@vagrant vagrant]hbase shell <<EOF create 't2','f2' put 't2', 'row1', 'f2:a', 'val1' scan 't2' EOF 14/01/07 22:36:54 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available create 't2','f2' 0 row(s) in 2.8070 seconds put 't2', 'row1', 'f2:a', 'val1' 0 row(s) in 0.1010 seconds scan 't2' ROW COLUMN+CELL row1 column=f2:a, timestamp=1389134219249, value=val1 1 row(s) in 0.0480 seconds Can we now commit this ? I think its ready.
        Hide
        jay vyas added a comment -
        • Evans this patch looks good. Im going to test it now !
        • Thanks for trying out fedora i understand why you needed to use centos. I will maybe try to put a fedora box that can also be used together.
        • for hbase-test.sh , should the scan also grep for the record you insert and retun non-zero if it fails?

        Will leave more feedback after i test it . Thanks again. This is going to make bigtop deployment FUN

        Show
        jay vyas added a comment - Evans this patch looks good. Im going to test it now ! Thanks for trying out fedora i understand why you needed to use centos. I will maybe try to put a fedora box that can also be used together. for hbase-test.sh , should the scan also grep for the record you insert and retun non-zero if it fails? Will leave more feedback after i test it . Thanks again. This is going to make bigtop deployment FUN
        Hide
        Evans Ye added a comment - - edited

        Hi Jay, good news!
        I've work out the first version of the patch. Please allow me to have a brief introduce to it:
        1) In this patch, hadoop and hbase will be deployed on a centos 6 VM using vagrant's puppet provisioner and bigtop's puppet recipes.
        2) The puppet recipes located on host machine will be configured to be a shared folder in guest VM to retrieve the configuration file.
        3) You can add or remove components easily through configuration file which is located at bigtop-deploy/puppet/config/site.csv on the host machine.
        4) The base VM box is provided by puppetlabs.

        One thing to let you know is that I was trying to work out the patch based on fedora19 box (same one with BIGTOP-1072),
        but the official puppetlabs' repo only provides puppet 3.x for that OS version.
        To let our puppet recipes support puppet 3, we need to get BIGTOP-1047 patched, but there should be a backward compatibility issue exist.

        Another thing is that I found there's a bug on the current puppet recipes: BIGTOP-1174, so if you would like to try the patch out, you may need to execute the puppet provisioner again(vagrant provision --provision-with puppet) after vagrant up.

        Show
        Evans Ye added a comment - - edited Hi Jay, good news! I've work out the first version of the patch. Please allow me to have a brief introduce to it: 1) In this patch, hadoop and hbase will be deployed on a centos 6 VM using vagrant's puppet provisioner and bigtop's puppet recipes. 2) The puppet recipes located on host machine will be configured to be a shared folder in guest VM to retrieve the configuration file. 3) You can add or remove components easily through configuration file which is located at bigtop-deploy/puppet/config/site.csv on the host machine. 4) The base VM box is provided by puppetlabs . One thing to let you know is that I was trying to work out the patch based on fedora19 box (same one with BIGTOP-1072 ), but the official puppetlabs' repo only provides puppet 3.x for that OS version. To let our puppet recipes support puppet 3, we need to get BIGTOP-1047 patched, but there should be a backward compatibility issue exist. Another thing is that I found there's a bug on the current puppet recipes: BIGTOP-1174 , so if you would like to try the patch out, you may need to execute the puppet provisioner again( vagrant provision --provision-with puppet ) after vagrant up .
        Hide
        Evans Ye added a comment -

        Thanks Jay, those suggestions are really good. I'll organize my work and provide a patch in these days

        Show
        Evans Ye added a comment - Thanks Jay, those suggestions are really good. I'll organize my work and provide a patch in these days
        Hide
        jay vyas added a comment -

        Hi Evans. I dont have a patch. Hereis what I was thinking...........

        1) use the native vagrant puppet support to provision the cluster (obviously). Maybe the puppet-deploy recipes in the bigtop/ directory can be included as a vagrant shared folder?

        2) Put the configuration file defining the conf components (just patched by roman here https://issues.apache.org/jira/browse/BIGTOP-1161) again in a shared folder, so that host can update the cluster configuration, and then "vagrant up" provisions the system accordingly.

        But i dont have a patch, would love to try yours out ! Let me know thanks!

        Show
        jay vyas added a comment - Hi Evans. I dont have a patch. Hereis what I was thinking........... 1) use the native vagrant puppet support to provision the cluster (obviously). Maybe the puppet-deploy recipes in the bigtop/ directory can be included as a vagrant shared folder? 2) Put the configuration file defining the conf components (just patched by roman here https://issues.apache.org/jira/browse/BIGTOP-1161 ) again in a shared folder, so that host can update the cluster configuration, and then "vagrant up" provisions the system accordingly. But i dont have a patch, would love to try yours out ! Let me know thanks!
        Hide
        Evans Ye added a comment -

        Hi Jay, I've done some work which is highly related to this jira, so if you're not yet started to deal with this, would you mind me to provide a concept patch for this? If you're going to do it, I can still help on the test

        Show
        Evans Ye added a comment - Hi Jay, I've done some work which is highly related to this jira, so if you're not yet started to deal with this, would you mind me to provide a concept patch for this? If you're going to do it, I can still help on the test

          People

          • Assignee:
            Evans Ye
            Reporter:
            jay vyas
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development