Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1627

Move enable_local_yum implementation to bigtop-deploy/utils/setup-env.sh to share with Docker provisioner

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 0.8.0
    • 1.0.0
    • None

    Description

      Currently, we have enable_local_yum feature implemented in vagrant-puppet-vm's Vagrantfile(see BIGTOP-1562's patch). This jira is proposing to move the implementation to bigtop-deploy/utils/setup-env.sh so that the we can share the general implementation between BigTop's VM provisoner and Docker provisioner.

      Attachments

        1. BIGTOP-1627.1.patch
          5 kB
          Evans Ye
        2. BIGTOP-1627.2.patch
          6 kB
          Evans Ye
        3. BIGTOP-1627.3.patch
          7 kB
          Evans Ye
        4. BIGTOP-1627.4.patch
          8 kB
          Evans Ye
        5. BIGTOP-1627.5.patch
          8 kB
          Evans Ye

        Activity

          jayunit100 jay vyas added a comment -

          Yup definitely we need this.
          veey important to keep docker features in sync w vm features where possible!

          jayunit100 jay vyas added a comment - Yup definitely we need this. veey important to keep docker features in sync w vm features where possible!
          evans_ye Evans Ye added a comment -

          This jira is primary to rearrange the implementation without breaking the feature in BigTop VM provisioner.
          I'll upload a patch to address this, then we can move forward to add the feature in to Docker provisioner.

          evans_ye Evans Ye added a comment - This jira is primary to rearrange the implementation without breaking the feature in BigTop VM provisioner. I'll upload a patch to address this, then we can move forward to add the feature in to Docker provisioner.
          jayunit100 jay vyas added a comment -

          evans_ye i think to save time, why not merge BIGTOP-1628 with this one? Its basically all one feature, i think. otherwise, makes sense

          jayunit100 jay vyas added a comment - evans_ye i think to save time, why not merge BIGTOP-1628 with this one? Its basically all one feature, i think. otherwise, makes sense
          evans_ye Evans Ye added a comment -

          Why not! I was thinking that there might be another way to put the feature in docker provisioner, but so far it seems reasonable to do it in one commit. I'll consolidate my works in one patch here

          evans_ye Evans Ye added a comment - Why not! I was thinking that there might be another way to put the feature in docker provisioner, but so far it seems reasonable to do it in one commit. I'll consolidate my works in one patch here
          evans_ye Evans Ye added a comment -

          Patch attached!
          Tested on both VM and Docker provisioner w/ smoke-tests enabled.
          FYR, It took around 70mins from zero to tests finished on my desktop.

          evans_ye Evans Ye added a comment - Patch attached! Tested on both VM and Docker provisioner w/ smoke-tests enabled. FYR, It took around 70mins from zero to tests finished on my desktop.
          jayunit100 jay vyas added a comment -

          70 mins? just for the smoke tests? Tjhats funny. well +1 it seems to look okay. why did you move the for loop BTW?

          jayunit100 jay vyas added a comment - 70 mins? just for the smoke tests? Tjhats funny. well +1 it seems to look okay. why did you move the for loop BTW?
          evans_ye Evans Ye added a comment -

          Hey jayunit100, 70mins is the round-trip time from cluster deployment to smoke-tests. Usually it tooks around 30 mins to get a cluster deployed, so the smoke-tests should be spending 40mins to complete.

          The change on for loop is a improvement to slightly reduce code duplication:

          echo "/bigtop-home/bigtop-deploy/vm/utils/setup-env.sh" |vagrant ssh ${nodes[$i]}		
          echo "/vagrant/provision.sh $hadoop_head_node $repo $components" |vagrant ssh ${nodes[$i]}
          

          And do environment setup in parallel mode as well.
          Because we need to make sure master nodes are up before slaves, I dedicate this in another loop.

          evans_ye Evans Ye added a comment - Hey jayunit100 , 70mins is the round-trip time from cluster deployment to smoke-tests. Usually it tooks around 30 mins to get a cluster deployed, so the smoke-tests should be spending 40mins to complete. The change on for loop is a improvement to slightly reduce code duplication: echo "/bigtop-home/bigtop-deploy/vm/utils/setup-env.sh" |vagrant ssh ${nodes[$i]} echo "/vagrant/provision.sh $hadoop_head_node $repo $components" |vagrant ssh ${nodes[$i]} And do environment setup in parallel mode as well. Because we need to make sure master nodes are up before slaves, I dedicate this in another loop.
          evans_ye Evans Ye added a comment - - edited

          Upload patch 2 to fix a bug on shell array representation:

          $nodes --> ${nodes[*]}
          
          evans_ye Evans Ye added a comment - - edited Upload patch 2 to fix a bug on shell array representation: $nodes --> ${nodes[*]}
          evans_ye Evans Ye added a comment -

          Attached a new patch reflect to the changes in BIGTOP-1643 and BIGTOP-1634.
          Tested and works well on centos container as well as virtualbox VM, not being able to test on debian.
          In the patch I also fix a stupid capital typo in docker provision.sh...The rest are all about enable_local_yum.

          evans_ye Evans Ye added a comment - Attached a new patch reflect to the changes in BIGTOP-1643 and BIGTOP-1634 . Tested and works well on centos container as well as virtualbox VM, not being able to test on debian. In the patch I also fix a stupid capital typo in docker provision.sh...The rest are all about enable_local_yum.
          oflebbe Olaf Flebbe added a comment -

          Thank you very much for looking into this!!!

          Tested on debian. Did not work since apt usually forces repositories have to be signed. I signed my bigtop repo and the gpg signature check fails, because of the missing key. See below for error message.

          So either we have to support signed repositories or we have to force apt not to check signatures.
          IMHO it wold be best to use the 'apt' puppet class to set up the apt location and supply a 'disable_keys' equals true parameter.
          Too late tonight to give a concrete patch now...

          Instructions to create a signed repo:

          The repo can be signed with this patch:

          diff --git a/bigtop-repos/apt/distributions b/bigtop-repos/apt/distributions
          index 470bb7e..b9c547a 100644
          --- a/bigtop-repos/apt/distributions
          +++ b/bigtop-repos/apt/distributions
          @@ -6,3 +6,4 @@ Version: 0.1
           Architectures: i386 amd64 source
           Components: contrib 
           Description: Bigtop
          +SignWith: default
          

          There must be a gpg key "default" in your keychain for this to work.

          The
          Somehow the vm needs access to the exported public gpg key (for instance named keys.asc) and issue the command
          "apt-key add key.asc" For instance we could use the convention to distribute the key in the repository itself at its root level.

          ...
          Debug: Executing '/usr/bin/dpkg-query -W --showformat '${Status} ${Package} ${Version}\n' hadoop-hdfs-datanode'
          Debug: Executing '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install hadoop-hdfs-datanode'
          Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install hadoop-hdfs-datanode' returned 100: Reading package lists...
          Building dependency tree...
          Reading state information...
          The following extra packages will be installed:
            bigtop-jsvc bigtop-utils hadoop hadoop-hdfs libjline-java liblog4j1.2-java
            libnetty-java libservlet2.5-java libslf4j-java libxerces2-java
            libxml-commons-external-java libxml-commons-resolver1.1-java
            libzookeeper-java libzookeeper-java-doc netcat-openbsd zookeeper
          Suggested packages:
            libjline-java-doc libmail-java liblog4j1.2-java-doc libcommons-logging-java
            libxerces2-java-doc libxerces2-java-gcj libxml-commons-resolver1.1-java-doc
          The following NEW packages will be installed:
            bigtop-jsvc bigtop-utils hadoop hadoop-hdfs hadoop-hdfs-datanode
            libjline-java liblog4j1.2-java libnetty-java libservlet2.5-java
            libslf4j-java libxerces2-java libxml-commons-external-java
            libxml-commons-resolver1.1-java libzookeeper-java libzookeeper-java-doc
            netcat-openbsd zookeeper
          0 upgraded, 17 newly installed, 0 to remove and 14 not upgraded.
          Need to get 4975 kB/40.9 MB of archives.
          After this operation, 59.5 MB of additional disk space will be used.
          WARNING: The following packages cannot be authenticated!
            bigtop-utils bigtop-jsvc hadoop hadoop-hdfs hadoop-hdfs-datanode
          E: There are problems and -y was used without --force-yes
          
          oflebbe Olaf Flebbe added a comment - Thank you very much for looking into this!!! Tested on debian. Did not work since apt usually forces repositories have to be signed. I signed my bigtop repo and the gpg signature check fails, because of the missing key. See below for error message. So either we have to support signed repositories or we have to force apt not to check signatures. IMHO it wold be best to use the 'apt' puppet class to set up the apt location and supply a 'disable_keys' equals true parameter. Too late tonight to give a concrete patch now... Instructions to create a signed repo: The repo can be signed with this patch: diff --git a/bigtop-repos/apt/distributions b/bigtop-repos/apt/distributions index 470bb7e..b9c547a 100644 --- a/bigtop-repos/apt/distributions +++ b/bigtop-repos/apt/distributions @@ -6,3 +6,4 @@ Version: 0.1 Architectures: i386 amd64 source Components: contrib Description: Bigtop +SignWith: default There must be a gpg key "default" in your keychain for this to work. The Somehow the vm needs access to the exported public gpg key (for instance named keys.asc) and issue the command "apt-key add key.asc" For instance we could use the convention to distribute the key in the repository itself at its root level. ... Debug: Executing '/usr/bin/dpkg-query -W --showformat ' ${Status} ${Package} ${Version}\n ' hadoop-hdfs-datanode' Debug: Executing '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install hadoop-hdfs-datanode' Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install hadoop-hdfs-datanode' returned 100: Reading package lists... Building dependency tree... Reading state information... The following extra packages will be installed: bigtop-jsvc bigtop-utils hadoop hadoop-hdfs libjline-java liblog4j1.2-java libnetty-java libservlet2.5-java libslf4j-java libxerces2-java libxml-commons-external-java libxml-commons-resolver1.1-java libzookeeper-java libzookeeper-java-doc netcat-openbsd zookeeper Suggested packages: libjline-java-doc libmail-java liblog4j1.2-java-doc libcommons-logging-java libxerces2-java-doc libxerces2-java-gcj libxml-commons-resolver1.1-java-doc The following NEW packages will be installed: bigtop-jsvc bigtop-utils hadoop hadoop-hdfs hadoop-hdfs-datanode libjline-java liblog4j1.2-java libnetty-java libservlet2.5-java libslf4j-java libxerces2-java libxml-commons-external-java libxml-commons-resolver1.1-java libzookeeper-java libzookeeper-java-doc netcat-openbsd zookeeper 0 upgraded, 17 newly installed, 0 to remove and 14 not upgraded. Need to get 4975 kB/40.9 MB of archives. After this operation, 59.5 MB of additional disk space will be used. WARNING: The following packages cannot be authenticated! bigtop-utils bigtop-jsvc hadoop hadoop-hdfs hadoop-hdfs-datanode E: There are problems and -y was used without --force-yes
          oflebbe Olaf Flebbe added a comment -

          Ah and by the way. Debian usually fails because there is a conflicting "zookeeper" package in debian. We have to pin our repository. We did this also with puppet...

          oflebbe Olaf Flebbe added a comment - Ah and by the way. Debian usually fails because there is a conflicting "zookeeper" package in debian. We have to pin our repository. We did this also with puppet...
          evans_ye Evans Ye added a comment -

          Hi oflebbe, thanks for reviewing this.
          You're right. The apt key check prevent the deployment to succeed.
          I've implement the apt repo auto-deploy in BIGTOP-1676 which disable the key check globally.
          After BIGTOP-1676 has been applied, this should work.
          Upload patch 4 which rename enable_local_yum to enable_local_repo, making the variable name more general.

          evans_ye Evans Ye added a comment - Hi oflebbe , thanks for reviewing this. You're right. The apt key check prevent the deployment to succeed. I've implement the apt repo auto-deploy in BIGTOP-1676 which disable the key check globally. After BIGTOP-1676 has been applied, this should work. Upload patch 4 which rename enable_local_yum to enable_local_repo, making the variable name more general.
          evans_ye Evans Ye added a comment -

          Upload patch 5 which re-based on top of BIGTOP-1676.

          evans_ye Evans Ye added a comment - Upload patch 5 which re-based on top of BIGTOP-1676 .

          Perhaps should be discussed elsewhere, but just today I found myself in the situation where I needed to create a node with {[docker-hadoop.sh}} script and install just HDFS component from 0.8 release, and then compliment it with another freshly built component's package. For the latter I had to manually create and put in place the repo file, copy the repodata/packages into the docker's base image, etc.

          I wonder if this possible to be able to have an array of repos in bigtop_repo_uri? This will ease the deployment for developers. Unfortunately, I am not that prolific with Puppet to do it in a couple of minutes. I am not even sure if that can be done?

          cos Konstantin I Boudnik added a comment - Perhaps should be discussed elsewhere, but just today I found myself in the situation where I needed to create a node with {[docker-hadoop.sh}} script and install just HDFS component from 0.8 release, and then compliment it with another freshly built component's package. For the latter I had to manually create and put in place the repo file, copy the repodata/packages into the docker's base image, etc. I wonder if this possible to be able to have an array of repos in bigtop_repo_uri? This will ease the deployment for developers. Unfortunately, I am not that prolific with Puppet to do it in a couple of minutes. I am not even sure if that can be done?
          evans_ye Evans Ye added a comment -

          Indeed that would be a great feature to have, and would be a much elegant, bolder solution than this one. We probably need to loop over the array if puppet's native resource does not support. From my point of view, can we have this at least work version in and then head to the uri array attempting?

          evans_ye Evans Ye added a comment - Indeed that would be a great feature to have, and would be a much elegant, bolder solution than this one. We probably need to loop over the array if puppet's native resource does not support. From my point of view, can we have this at least work version in and then head to the uri array attempting?
          cos Konstantin I Boudnik added a comment - - edited

          By all means - I didn't intend to stretch this JIRA to the multiple repos specs. Sorry for stealing the thread.

          cos Konstantin I Boudnik added a comment - - edited By all means - I didn't intend to stretch this JIRA to the multiple repos specs. Sorry for stealing the thread.
          jayunit100 jay vyas added a comment - - edited

          +1 but untested I figure lets get it in because it's blocking progress on docker work !

          Please test it good before pushing. Code looks good to me!

          jayunit100 jay vyas added a comment - - edited +1 but untested I figure lets get it in because it's blocking progress on docker work ! Please test it good before pushing. Code looks good to me!
          evans_ye Evans Ye added a comment -

          Thanks, let me test this with newest git repo.

          evans_ye Evans Ye added a comment - Thanks, let me test this with newest git repo.
          evans_ye Evans Ye added a comment -

          I've re-test this on centos and debian, both works well.
          Committed. Thanks for the review jayunit100.

          evans_ye Evans Ye added a comment - I've re-test this on centos and debian, both works well. Committed. Thanks for the review jayunit100 .

          People

            evans_ye Evans Ye
            evans_ye Evans Ye
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: