Since tez require libraries to be uploaded to hdfs, I add the upload logic into init-hcfs.json and init-hcfs.groovy.
The patch also added tez conf into hadoop-env.sh, set hive.execution.engine to tez, and set mapreduce framework to yarn-tez when tez is deployed.
This makes the testing of this patch a little bit complicated. Here's the steps test I used:
- ./gradlew hadoop-yum bigtop-utils-yum to build new hadoop and bigtop-utils after patch applied
- update the bigtop-deploy/vm/vagrant-puppet-docker/vagrantconfig.yaml:
components: [hadoop, yarn, hive, tez]
- cd bigtop-deploy/vm/vagrant-puppet-docekr; ./docker-hadoop -d -c 1 (takes 4mins in my environment)
- vagrant ssh bigtop1
- su - jenkins
- hadoop fs -put /etc/passwd
- hive (get into the hive shell)
- create table t1 (s string) location '/user/jenkins'
- select * from t1 order by s;
- Should see a fancy tez execution table