Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1449

Add RC Testing support to vagrant recipe

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 1.0.0
    • Component/s: vm
    • Labels:
      None

      Description

      We can automatically set up vagrant to install a release candidate on n nodes and run smoke tests. Here is how .

      1) Adding the bigtop folder as a shared folder in Vagrantfile, so that VMs have access to bigtop source

         # the bigtop puppet recipes
         bigtop_puppet = "../../puppet"
      -
      +  bigtop_home =   "../../../"
      ...
      +  bigtop.vm.synced_folder bigtop_home, "/bigtop-home"
       

      2) cd /bigtop-home [ sudo puppet apply --modulepath=./ -e "include bigtop_toolchain::gradle && alias gradle=/usr/local/gradle/bin/gradle"

      3) Having the bigtop repo in Vagrantfile point at the RC url (modify the line in provision.sh bigtop_yumrepo_uri,http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.8.0/label=centos6/6/artifact/output/

      4) Export HADOOP_CONF_DIR, BIGTOP_HOME,HADOOP_MAPRED_HOME,HIVE_HOME,HIVE_CONF_DIR, and so on...

      5) Ensure that a home dir exists for the user (i.e. as fs superuser (i.e. hdfs or root on a posix filesystem) run hadoop fs -mkdir /user/vagrant )

      6) Running the tests cd bigtop-tests/smoke-tests/ && gradle compileGroovy test -Dsmoke-tests=mapreduce --info

      Might as well do it so that for next release, we can automate testing of bigtop on n-node clusters.

      When doing this task,

      • lets also update smoke-tests and (1) fix the README (it sais smoke-tests, instead of smoke.tests) (2) parameterize ITest . Those are both needed for proper automation of RC testing.
      • Also lets bound the amount of hive tests we run. Currently it runs all of them. I think we only need it to run "basic" or a few others, and we can ensure this by setting system properties like this, so the smoke tests is fast and simple:
         
        test {
         systemProperties["org.apache.bigtop.itest.hivesmoke.TestHiveSmokeBulk.test_include"]="basic"
        }
        

      So all in all there are basically two tasks: Add a "test.sh" script to the vagrant provisioner which installs toolchain and runs bigtop smoke tests, and a few complimentary, minor updates to the smoke-tests .

      Then, next time an RC comes out, to test, we just update number of nodes and run "vagrant up".

      1. BIGTOP-1449.patch
        8 kB
        jay vyas
      2. BIGTOP-1499.patch
        8 kB
        jay vyas

        Activity

        Hide
        jayunit100 jay vyas added a comment -

        okay , commited !

        Show
        jayunit100 jay vyas added a comment - okay , commited !
        Hide
        jayunit100 jay vyas added a comment -

        okay, thanks Evans Ye ! fixed the commit message and commiting now.
        I agree 3 nodes is better, we can make the vagrant infra completely customizable by config file in a follow up JIRA.

        Show
        jayunit100 jay vyas added a comment - okay, thanks Evans Ye ! fixed the commit message and commiting now. I agree 3 nodes is better, we can make the vagrant infra completely customizable by config file in a follow up JIRA.
        Hide
        evans_ye Evans Ye added a comment -

        The patch ran successfully on both 1 node and 3 nodes cluster.
        Just some minor notes here:

        • The commit message is BIGTOP-1499 instead of 1449.
        • Are you intended to make the num_instances default to 1? I'm just thinking that makes it greater than 1 best demonstrate its power.

        I think the patch is already good enough.
        Since those concerns are not related to the functionality of this patch, maybe you can just have some little fixes and then it will be ready to be committed.
        In brief, +1 on this patch!

        Show
        evans_ye Evans Ye added a comment - The patch ran successfully on both 1 node and 3 nodes cluster. Just some minor notes here: The commit message is BIGTOP-1499 instead of 1449. Are you intended to make the num_instances default to 1? I'm just thinking that makes it greater than 1 best demonstrate its power. I think the patch is already good enough. Since those concerns are not related to the functionality of this patch, maybe you can just have some little fixes and then it will be ready to be committed. In brief, +1 on this patch!
        Hide
        jayunit100 jay vyas added a comment - - edited

        okay ! heres the final auto release tester. Jon Cope or Evans Ye can you review this again, and +1 or -1 it for commit ? Thanks !!!!!!!

        Show
        jayunit100 jay vyas added a comment - - edited okay ! heres the final auto release tester. Jon Cope or Evans Ye can you review this again, and +1 or -1 it for commit ? Thanks !!!!!!!
        Hide
        jayunit100 jay vyas added a comment -

        Thanks jon for the fixes. Should be easy for me to roll those in. I'll submit the updated patch and then we will have an auto release tester

        Show
        jayunit100 jay vyas added a comment - Thanks jon for the fixes. Should be easy for me to roll those in. I'll submit the updated patch and then we will have an auto release tester
        Hide
        jcope Jon Cope added a comment - - edited

        I've validated the smoke-tests.sh with BT-1449 applied. The script just needs a few small tweaks to smooth it out:

        Mapred:

        • Err: `java.lang.AssertionError: Can't find hadoop-examples.jar file`
        • FIX: smoke-tests.sh: HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce/ from ".../hadoop-mapred"

        Hive:

        • Success! no changes

        Pig:

        • Err: undeclared env variable: PIG_HOME
        • FIX: add export PIG_HOME=/usr/lib/pig" to smoke-test.sh

        Flume:

        • add flume to yum install list
        • add FLUME_HOME env var

        Sqoop:

        • same as flume

        Other than that, it ran smoothly!

        edit: formatting

        Show
        jcope Jon Cope added a comment - - edited I've validated the smoke-tests.sh with BT-1449 applied. The script just needs a few small tweaks to smooth it out: Mapred: Err: `java.lang.AssertionError: Can't find hadoop-examples.jar file` FIX: smoke-tests.sh: HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce/ from ".../hadoop-mapred" Hive: Success! no changes Pig: Err: undeclared env variable: PIG_HOME FIX: add export PIG_HOME=/usr/lib/pig" to smoke-test.sh Flume: add flume to yum install list add FLUME_HOME env var Sqoop: same as flume Other than that, it ran smoothly! edit: formatting
        Hide
        jayunit100 jay vyas added a comment -

        Heres a patch that automates this. there might be a couple minor tweaks but its pretty much completed .

        I'll put some finishing touches on it tomorrow and then we can merge it in for release testing from now on.

        Show
        jayunit100 jay vyas added a comment - Heres a patch that automates this. there might be a couple minor tweaks but its pretty much completed . I'll put some finishing touches on it tomorrow and then we can merge it in for release testing from now on.
        Hide
        jayunit100 jay vyas added a comment -

        i now have autopmated this as part of the frenzy to figure out WTH is going on in BIGTOP-1470 (mahout in RC 0.8). testing now - once its working will attach a patch.

        Show
        jayunit100 jay vyas added a comment - i now have autopmated this as part of the frenzy to figure out WTH is going on in BIGTOP-1470 (mahout in RC 0.8). testing now - once its working will attach a patch.
        Hide
        rvs Roman Shaposhnik added a comment -

        As a matter of fact, if all the steps above could be scripted – that would be super useful to have!

        Show
        rvs Roman Shaposhnik added a comment - As a matter of fact, if all the steps above could be scripted – that would be super useful to have!

          People

          • Assignee:
            jayunit100 jay vyas
            Reporter:
            jayunit100 jay vyas
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development