Major Description
We can automatically set up vagrant to install a release candidate on n nodes and run smoke tests. Here is how .
1) Adding the bigtop folder as a shared folder in Vagrantfile, so that VMs have access to bigtop source
# the bigtop puppet recipes bigtop_puppet = "../../puppet" - + bigtop_home = "../../../" ... + bigtop.vm.synced_folder bigtop_home, "/bigtop-home"
2) cd /bigtop-home [ sudo puppet apply --modulepath=./ -e "include bigtop_toolchain::gradle && alias gradle=/usr/local/gradle/bin/gradle"
3) Having the bigtop repo in Vagrantfile point at the RC url (modify the line in provision.sh bigtop_yumrepo_uri,http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.8.0/label=centos6/6/artifact/output/
4) Export HADOOP_CONF_DIR, BIGTOP_HOME,HADOOP_MAPRED_HOME,HIVE_HOME,HIVE_CONF_DIR, and so on...
5) Ensure that a home dir exists for the user (i.e. as fs superuser (i.e. hdfs or root on a posix filesystem) run hadoop fs -mkdir /user/vagrant )
6) Running the tests cd bigtop-tests/smoke-tests/ && gradle compileGroovy test -Dsmoke-tests=mapreduce --info
Might as well do it so that for next release, we can automate testing of bigtop on n-node clusters.
When doing this task,
- lets also update smoke-tests and (1) fix the README (it sais smoke-tests, instead of smoke.tests) (2) parameterize ITest . Those are both needed for proper automation of RC testing.
- Also lets bound the amount of hive tests we run. Currently it runs all of them. I think we only need it to run "basic" or a few others, and we can ensure this by setting system properties like this, so the smoke tests is fast and simple:
test { systemProperties["org.apache.bigtop.itest.hivesmoke.TestHiveSmokeBulk.test_include"]="basic" }
So all in all there are basically two tasks: Add a "test.sh" script to the vagrant provisioner which installs toolchain and runs bigtop smoke tests, and a few complimentary, minor updates to the smoke-tests .
Then, next time an RC comes out, to test, we just update number of nodes and run "vagrant up". 