Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
0.8.0
-
None
Description
Currently, we have enable_local_yum feature implemented in vagrant-puppet-vm's Vagrantfile(see BIGTOP-1562's patch). This jira is proposing to move the implementation to bigtop-deploy/utils/setup-env.sh so that the we can share the general implementation between BigTop's VM provisoner and Docker provisioner.
Attachments
Attachments
- BIGTOP-1627.1.patch
- 5 kB
- Evans Ye
- BIGTOP-1627.2.patch
- 6 kB
- Evans Ye
- BIGTOP-1627.3.patch
- 7 kB
- Evans Ye
- BIGTOP-1627.4.patch
- 8 kB
- Evans Ye
- BIGTOP-1627.5.patch
- 8 kB
- Evans Ye
Activity
This jira is primary to rearrange the implementation without breaking the feature in BigTop VM provisioner.
I'll upload a patch to address this, then we can move forward to add the feature in to Docker provisioner.
evans_ye i think to save time, why not merge BIGTOP-1628 with this one? Its basically all one feature, i think. otherwise, makes sense
Why not! I was thinking that there might be another way to put the feature in docker provisioner, but so far it seems reasonable to do it in one commit. I'll consolidate my works in one patch here
Patch attached!
Tested on both VM and Docker provisioner w/ smoke-tests enabled.
FYR, It took around 70mins from zero to tests finished on my desktop.
70 mins? just for the smoke tests? Tjhats funny. well +1 it seems to look okay. why did you move the for loop BTW?
Hey jayunit100, 70mins is the round-trip time from cluster deployment to smoke-tests. Usually it tooks around 30 mins to get a cluster deployed, so the smoke-tests should be spending 40mins to complete.
The change on for loop is a improvement to slightly reduce code duplication:
echo "/bigtop-home/bigtop-deploy/vm/utils/setup-env.sh" |vagrant ssh ${nodes[$i]} echo "/vagrant/provision.sh $hadoop_head_node $repo $components" |vagrant ssh ${nodes[$i]}
And do environment setup in parallel mode as well.
Because we need to make sure master nodes are up before slaves, I dedicate this in another loop.
Upload patch 2 to fix a bug on shell array representation:
$nodes --> ${nodes[*]}
Attached a new patch reflect to the changes in BIGTOP-1643 and BIGTOP-1634.
Tested and works well on centos container as well as virtualbox VM, not being able to test on debian.
In the patch I also fix a stupid capital typo in docker provision.sh...The rest are all about enable_local_yum.
Thank you very much for looking into this!!!
Tested on debian. Did not work since apt usually forces repositories have to be signed. I signed my bigtop repo and the gpg signature check fails, because of the missing key. See below for error message.
So either we have to support signed repositories or we have to force apt not to check signatures.
IMHO it wold be best to use the 'apt' puppet class to set up the apt location and supply a 'disable_keys' equals true parameter.
Too late tonight to give a concrete patch now...
Instructions to create a signed repo:
The repo can be signed with this patch:
diff --git a/bigtop-repos/apt/distributions b/bigtop-repos/apt/distributions
index 470bb7e..b9c547a 100644
--- a/bigtop-repos/apt/distributions
+++ b/bigtop-repos/apt/distributions
@@ -6,3 +6,4 @@ Version: 0.1
Architectures: i386 amd64 source
Components: contrib
Description: Bigtop
+SignWith: default
There must be a gpg key "default" in your keychain for this to work.
The
Somehow the vm needs access to the exported public gpg key (for instance named keys.asc) and issue the command
"apt-key add key.asc" For instance we could use the convention to distribute the key in the repository itself at its root level.
... Debug: Executing '/usr/bin/dpkg-query -W --showformat '${Status} ${Package} ${Version}\n' hadoop-hdfs-datanode' Debug: Executing '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install hadoop-hdfs-datanode' Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install hadoop-hdfs-datanode' returned 100: Reading package lists... Building dependency tree... Reading state information... The following extra packages will be installed: bigtop-jsvc bigtop-utils hadoop hadoop-hdfs libjline-java liblog4j1.2-java libnetty-java libservlet2.5-java libslf4j-java libxerces2-java libxml-commons-external-java libxml-commons-resolver1.1-java libzookeeper-java libzookeeper-java-doc netcat-openbsd zookeeper Suggested packages: libjline-java-doc libmail-java liblog4j1.2-java-doc libcommons-logging-java libxerces2-java-doc libxerces2-java-gcj libxml-commons-resolver1.1-java-doc The following NEW packages will be installed: bigtop-jsvc bigtop-utils hadoop hadoop-hdfs hadoop-hdfs-datanode libjline-java liblog4j1.2-java libnetty-java libservlet2.5-java libslf4j-java libxerces2-java libxml-commons-external-java libxml-commons-resolver1.1-java libzookeeper-java libzookeeper-java-doc netcat-openbsd zookeeper 0 upgraded, 17 newly installed, 0 to remove and 14 not upgraded. Need to get 4975 kB/40.9 MB of archives. After this operation, 59.5 MB of additional disk space will be used. WARNING: The following packages cannot be authenticated! bigtop-utils bigtop-jsvc hadoop hadoop-hdfs hadoop-hdfs-datanode E: There are problems and -y was used without --force-yes
Ah and by the way. Debian usually fails because there is a conflicting "zookeeper" package in debian. We have to pin our repository. We did this also with puppet...
Hi oflebbe, thanks for reviewing this.
You're right. The apt key check prevent the deployment to succeed.
I've implement the apt repo auto-deploy in BIGTOP-1676 which disable the key check globally.
After BIGTOP-1676 has been applied, this should work.
Upload patch 4 which rename enable_local_yum to enable_local_repo, making the variable name more general.
Perhaps should be discussed elsewhere, but just today I found myself in the situation where I needed to create a node with {[docker-hadoop.sh}} script and install just HDFS component from 0.8 release, and then compliment it with another freshly built component's package. For the latter I had to manually create and put in place the repo file, copy the repodata/packages into the docker's base image, etc.
I wonder if this possible to be able to have an array of repos in bigtop_repo_uri? This will ease the deployment for developers. Unfortunately, I am not that prolific with Puppet to do it in a couple of minutes. I am not even sure if that can be done?
Indeed that would be a great feature to have, and would be a much elegant, bolder solution than this one. We probably need to loop over the array if puppet's native resource does not support. From my point of view, can we have this at least work version in and then head to the uri array attempting?
By all means - I didn't intend to stretch this JIRA to the multiple repos specs. Sorry for stealing the thread.
+1 but untested I figure lets get it in because it's blocking progress on docker work !
Please test it good before pushing. Code looks good to me!
I've re-test this on centos and debian, both works well.
Committed. Thanks for the review jayunit100.
Yup definitely we need this.
veey important to keep docker features in sync w vm features where possible!