Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1449

Add RC Testing support to vagrant recipe

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 1.0.0
    • Component/s: vm
    • Labels:
      None

      Description

      We can automatically set up vagrant to install a release candidate on n nodes and run smoke tests. Here is how .

      1) Adding the bigtop folder as a shared folder in Vagrantfile, so that VMs have access to bigtop source

         # the bigtop puppet recipes
         bigtop_puppet = "../../puppet"
      -
      +  bigtop_home =   "../../../"
      ...
      +  bigtop.vm.synced_folder bigtop_home, "/bigtop-home"
       

      2) cd /bigtop-home [ sudo puppet apply --modulepath=./ -e "include bigtop_toolchain::gradle && alias gradle=/usr/local/gradle/bin/gradle"

      3) Having the bigtop repo in Vagrantfile point at the RC url (modify the line in provision.sh bigtop_yumrepo_uri,http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.8.0/label=centos6/6/artifact/output/

      4) Export HADOOP_CONF_DIR, BIGTOP_HOME,HADOOP_MAPRED_HOME,HIVE_HOME,HIVE_CONF_DIR, and so on...

      5) Ensure that a home dir exists for the user (i.e. as fs superuser (i.e. hdfs or root on a posix filesystem) run hadoop fs -mkdir /user/vagrant )

      6) Running the tests cd bigtop-tests/smoke-tests/ && gradle compileGroovy test -Dsmoke-tests=mapreduce --info

      Might as well do it so that for next release, we can automate testing of bigtop on n-node clusters.

      When doing this task,

      • lets also update smoke-tests and (1) fix the README (it sais smoke-tests, instead of smoke.tests) (2) parameterize ITest . Those are both needed for proper automation of RC testing.
      • Also lets bound the amount of hive tests we run. Currently it runs all of them. I think we only need it to run "basic" or a few others, and we can ensure this by setting system properties like this, so the smoke tests is fast and simple:
         
        test {
         systemProperties["org.apache.bigtop.itest.hivesmoke.TestHiveSmokeBulk.test_include"]="basic"
        }
        

      So all in all there are basically two tasks: Add a "test.sh" script to the vagrant provisioner which installs toolchain and runs bigtop smoke tests, and a few complimentary, minor updates to the smoke-tests .

      Then, next time an RC comes out, to test, we just update number of nodes and run "vagrant up".

        Attachments

        1. BIGTOP-1449.patch
          8 kB
          jay vyas
        2. BIGTOP-1499.patch
          8 kB
          jay vyas

          Activity

            People

            • Assignee:
              jayunit100 jay vyas
              Reporter:
              jayunit100 jay vyas
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: