Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1627

Move enable_local_yum implementation to bigtop-deploy/utils/setup-env.sh to share with Docker provisioner

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 1.0.0
    • Component/s: None
    • Labels:

      Description

      Currently, we have enable_local_yum feature implemented in vagrant-puppet-vm's Vagrantfile(see BIGTOP-1562's patch). This jira is proposing to move the implementation to bigtop-deploy/utils/setup-env.sh so that the we can share the general implementation between BigTop's VM provisoner and Docker provisioner.

      1. BIGTOP-1627.1.patch
        5 kB
        Evans Ye
      2. BIGTOP-1627.2.patch
        6 kB
        Evans Ye
      3. BIGTOP-1627.3.patch
        7 kB
        Evans Ye
      4. BIGTOP-1627.4.patch
        8 kB
        Evans Ye
      5. BIGTOP-1627.5.patch
        8 kB
        Evans Ye

        Activity

        Hide
        jayunit100 jay vyas added a comment -

        Yup definitely we need this.
        veey important to keep docker features in sync w vm features where possible!

        Show
        jayunit100 jay vyas added a comment - Yup definitely we need this. veey important to keep docker features in sync w vm features where possible!
        Hide
        evans_ye Evans Ye added a comment -

        This jira is primary to rearrange the implementation without breaking the feature in BigTop VM provisioner.
        I'll upload a patch to address this, then we can move forward to add the feature in to Docker provisioner.

        Show
        evans_ye Evans Ye added a comment - This jira is primary to rearrange the implementation without breaking the feature in BigTop VM provisioner. I'll upload a patch to address this, then we can move forward to add the feature in to Docker provisioner.
        Hide
        jayunit100 jay vyas added a comment -

        Evans Ye i think to save time, why not merge BIGTOP-1628 with this one? Its basically all one feature, i think. otherwise, makes sense

        Show
        jayunit100 jay vyas added a comment - Evans Ye i think to save time, why not merge BIGTOP-1628 with this one? Its basically all one feature, i think. otherwise, makes sense
        Hide
        evans_ye Evans Ye added a comment -

        Why not! I was thinking that there might be another way to put the feature in docker provisioner, but so far it seems reasonable to do it in one commit. I'll consolidate my works in one patch here

        Show
        evans_ye Evans Ye added a comment - Why not! I was thinking that there might be another way to put the feature in docker provisioner, but so far it seems reasonable to do it in one commit. I'll consolidate my works in one patch here
        Hide
        evans_ye Evans Ye added a comment -

        Patch attached!
        Tested on both VM and Docker provisioner w/ smoke-tests enabled.
        FYR, It took around 70mins from zero to tests finished on my desktop.

        Show
        evans_ye Evans Ye added a comment - Patch attached! Tested on both VM and Docker provisioner w/ smoke-tests enabled. FYR, It took around 70mins from zero to tests finished on my desktop.
        Hide
        jayunit100 jay vyas added a comment -

        70 mins? just for the smoke tests? Tjhats funny. well +1 it seems to look okay. why did you move the for loop BTW?

        Show
        jayunit100 jay vyas added a comment - 70 mins? just for the smoke tests? Tjhats funny. well +1 it seems to look okay. why did you move the for loop BTW?
        Hide
        evans_ye Evans Ye added a comment -

        Hey jay vyas, 70mins is the round-trip time from cluster deployment to smoke-tests. Usually it tooks around 30 mins to get a cluster deployed, so the smoke-tests should be spending 40mins to complete.

        The change on for loop is a improvement to slightly reduce code duplication:

        echo "/bigtop-home/bigtop-deploy/vm/utils/setup-env.sh" |vagrant ssh ${nodes[$i]}		
        echo "/vagrant/provision.sh $hadoop_head_node $repo $components" |vagrant ssh ${nodes[$i]}
        

        And do environment setup in parallel mode as well.
        Because we need to make sure master nodes are up before slaves, I dedicate this in another loop.

        Show
        evans_ye Evans Ye added a comment - Hey jay vyas , 70mins is the round-trip time from cluster deployment to smoke-tests. Usually it tooks around 30 mins to get a cluster deployed, so the smoke-tests should be spending 40mins to complete. The change on for loop is a improvement to slightly reduce code duplication: echo "/bigtop-home/bigtop-deploy/vm/utils/setup-env.sh" |vagrant ssh ${nodes[$i]} echo "/vagrant/provision.sh $hadoop_head_node $repo $components" |vagrant ssh ${nodes[$i]} And do environment setup in parallel mode as well. Because we need to make sure master nodes are up before slaves, I dedicate this in another loop.
        Hide
        evans_ye Evans Ye added a comment - - edited

        Upload patch 2 to fix a bug on shell array representation:

        $nodes --> ${nodes[*]}
        
        Show
        evans_ye Evans Ye added a comment - - edited Upload patch 2 to fix a bug on shell array representation: $nodes --> ${nodes[*]}
        Hide
        evans_ye Evans Ye added a comment -

        Attached a new patch reflect to the changes in BIGTOP-1643 and BIGTOP-1634.
        Tested and works well on centos container as well as virtualbox VM, not being able to test on debian.
        In the patch I also fix a stupid capital typo in docker provision.sh...The rest are all about enable_local_yum.

        Show
        evans_ye Evans Ye added a comment - Attached a new patch reflect to the changes in BIGTOP-1643 and BIGTOP-1634 . Tested and works well on centos container as well as virtualbox VM, not being able to test on debian. In the patch I also fix a stupid capital typo in docker provision.sh...The rest are all about enable_local_yum.
        Hide
        oflebbe Olaf Flebbe added a comment -

        Thank you very much for looking into this!!!

        Tested on debian. Did not work since apt usually forces repositories have to be signed. I signed my bigtop repo and the gpg signature check fails, because of the missing key. See below for error message.

        So either we have to support signed repositories or we have to force apt not to check signatures.
        IMHO it wold be best to use the 'apt' puppet class to set up the apt location and supply a 'disable_keys' equals true parameter.
        Too late tonight to give a concrete patch now...

        Instructions to create a signed repo:

        The repo can be signed with this patch:

        diff --git a/bigtop-repos/apt/distributions b/bigtop-repos/apt/distributions
        index 470bb7e..b9c547a 100644
        --- a/bigtop-repos/apt/distributions
        +++ b/bigtop-repos/apt/distributions
        @@ -6,3 +6,4 @@ Version: 0.1
         Architectures: i386 amd64 source
         Components: contrib 
         Description: Bigtop
        +SignWith: default
        

        There must be a gpg key "default" in your keychain for this to work.

        The
        Somehow the vm needs access to the exported public gpg key (for instance named keys.asc) and issue the command
        "apt-key add key.asc" For instance we could use the convention to distribute the key in the repository itself at its root level.

        ...
        Debug: Executing '/usr/bin/dpkg-query -W --showformat '${Status} ${Package} ${Version}\n' hadoop-hdfs-datanode'
        Debug: Executing '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install hadoop-hdfs-datanode'
        Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install hadoop-hdfs-datanode' returned 100: Reading package lists...
        Building dependency tree...
        Reading state information...
        The following extra packages will be installed:
          bigtop-jsvc bigtop-utils hadoop hadoop-hdfs libjline-java liblog4j1.2-java
          libnetty-java libservlet2.5-java libslf4j-java libxerces2-java
          libxml-commons-external-java libxml-commons-resolver1.1-java
          libzookeeper-java libzookeeper-java-doc netcat-openbsd zookeeper
        Suggested packages:
          libjline-java-doc libmail-java liblog4j1.2-java-doc libcommons-logging-java
          libxerces2-java-doc libxerces2-java-gcj libxml-commons-resolver1.1-java-doc
        The following NEW packages will be installed:
          bigtop-jsvc bigtop-utils hadoop hadoop-hdfs hadoop-hdfs-datanode
          libjline-java liblog4j1.2-java libnetty-java libservlet2.5-java
          libslf4j-java libxerces2-java libxml-commons-external-java
          libxml-commons-resolver1.1-java libzookeeper-java libzookeeper-java-doc
          netcat-openbsd zookeeper
        0 upgraded, 17 newly installed, 0 to remove and 14 not upgraded.
        Need to get 4975 kB/40.9 MB of archives.
        After this operation, 59.5 MB of additional disk space will be used.
        WARNING: The following packages cannot be authenticated!
          bigtop-utils bigtop-jsvc hadoop hadoop-hdfs hadoop-hdfs-datanode
        E: There are problems and -y was used without --force-yes
        
        Show
        oflebbe Olaf Flebbe added a comment - Thank you very much for looking into this!!! Tested on debian. Did not work since apt usually forces repositories have to be signed. I signed my bigtop repo and the gpg signature check fails, because of the missing key. See below for error message. So either we have to support signed repositories or we have to force apt not to check signatures. IMHO it wold be best to use the 'apt' puppet class to set up the apt location and supply a 'disable_keys' equals true parameter. Too late tonight to give a concrete patch now... Instructions to create a signed repo: The repo can be signed with this patch: diff --git a/bigtop-repos/apt/distributions b/bigtop-repos/apt/distributions index 470bb7e..b9c547a 100644 --- a/bigtop-repos/apt/distributions +++ b/bigtop-repos/apt/distributions @@ -6,3 +6,4 @@ Version: 0.1 Architectures: i386 amd64 source Components: contrib Description: Bigtop +SignWith: default There must be a gpg key "default" in your keychain for this to work. The Somehow the vm needs access to the exported public gpg key (for instance named keys.asc) and issue the command "apt-key add key.asc" For instance we could use the convention to distribute the key in the repository itself at its root level. ... Debug: Executing '/usr/bin/dpkg-query -W --showformat '${Status} ${Package} ${Version}\n' hadoop-hdfs-datanode' Debug: Executing '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install hadoop-hdfs-datanode' Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install hadoop-hdfs-datanode' returned 100: Reading package lists... Building dependency tree... Reading state information... The following extra packages will be installed: bigtop-jsvc bigtop-utils hadoop hadoop-hdfs libjline-java liblog4j1.2-java libnetty-java libservlet2.5-java libslf4j-java libxerces2-java libxml-commons-external-java libxml-commons-resolver1.1-java libzookeeper-java libzookeeper-java-doc netcat-openbsd zookeeper Suggested packages: libjline-java-doc libmail-java liblog4j1.2-java-doc libcommons-logging-java libxerces2-java-doc libxerces2-java-gcj libxml-commons-resolver1.1-java-doc The following NEW packages will be installed: bigtop-jsvc bigtop-utils hadoop hadoop-hdfs hadoop-hdfs-datanode libjline-java liblog4j1.2-java libnetty-java libservlet2.5-java libslf4j-java libxerces2-java libxml-commons-external-java libxml-commons-resolver1.1-java libzookeeper-java libzookeeper-java-doc netcat-openbsd zookeeper 0 upgraded, 17 newly installed, 0 to remove and 14 not upgraded. Need to get 4975 kB/40.9 MB of archives. After this operation, 59.5 MB of additional disk space will be used. WARNING: The following packages cannot be authenticated! bigtop-utils bigtop-jsvc hadoop hadoop-hdfs hadoop-hdfs-datanode E: There are problems and -y was used without --force-yes
        Hide
        oflebbe Olaf Flebbe added a comment -

        Ah and by the way. Debian usually fails because there is a conflicting "zookeeper" package in debian. We have to pin our repository. We did this also with puppet...

        Show
        oflebbe Olaf Flebbe added a comment - Ah and by the way. Debian usually fails because there is a conflicting "zookeeper" package in debian. We have to pin our repository. We did this also with puppet...
        Hide
        evans_ye Evans Ye added a comment -

        Hi Olaf Flebbe, thanks for reviewing this.
        You're right. The apt key check prevent the deployment to succeed.
        I've implement the apt repo auto-deploy in BIGTOP-1676 which disable the key check globally.
        After BIGTOP-1676 has been applied, this should work.
        Upload patch 4 which rename enable_local_yum to enable_local_repo, making the variable name more general.

        Show
        evans_ye Evans Ye added a comment - Hi Olaf Flebbe , thanks for reviewing this. You're right. The apt key check prevent the deployment to succeed. I've implement the apt repo auto-deploy in BIGTOP-1676 which disable the key check globally. After BIGTOP-1676 has been applied, this should work. Upload patch 4 which rename enable_local_yum to enable_local_repo, making the variable name more general.
        Hide
        evans_ye Evans Ye added a comment -

        Upload patch 5 which re-based on top of BIGTOP-1676.

        Show
        evans_ye Evans Ye added a comment - Upload patch 5 which re-based on top of BIGTOP-1676 .
        Hide
        cos Konstantin Boudnik added a comment -

        Perhaps should be discussed elsewhere, but just today I found myself in the situation where I needed to create a node with {[docker-hadoop.sh}} script and install just HDFS component from 0.8 release, and then compliment it with another freshly built component's package. For the latter I had to manually create and put in place the repo file, copy the repodata/packages into the docker's base image, etc.

        I wonder if this possible to be able to have an array of repos in bigtop_repo_uri? This will ease the deployment for developers. Unfortunately, I am not that prolific with Puppet to do it in a couple of minutes. I am not even sure if that can be done?

        Show
        cos Konstantin Boudnik added a comment - Perhaps should be discussed elsewhere, but just today I found myself in the situation where I needed to create a node with {[docker-hadoop.sh}} script and install just HDFS component from 0.8 release, and then compliment it with another freshly built component's package. For the latter I had to manually create and put in place the repo file, copy the repodata/packages into the docker's base image, etc. I wonder if this possible to be able to have an array of repos in bigtop_repo_uri? This will ease the deployment for developers. Unfortunately, I am not that prolific with Puppet to do it in a couple of minutes. I am not even sure if that can be done?
        Hide
        evans_ye Evans Ye added a comment -

        Indeed that would be a great feature to have, and would be a much elegant, bolder solution than this one. We probably need to loop over the array if puppet's native resource does not support. From my point of view, can we have this at least work version in and then head to the uri array attempting?

        Show
        evans_ye Evans Ye added a comment - Indeed that would be a great feature to have, and would be a much elegant, bolder solution than this one. We probably need to loop over the array if puppet's native resource does not support. From my point of view, can we have this at least work version in and then head to the uri array attempting?
        Hide
        cos Konstantin Boudnik added a comment - - edited

        By all means - I didn't intend to stretch this JIRA to the multiple repos specs. Sorry for stealing the thread.

        Show
        cos Konstantin Boudnik added a comment - - edited By all means - I didn't intend to stretch this JIRA to the multiple repos specs. Sorry for stealing the thread.
        Hide
        jayunit100 jay vyas added a comment - - edited

        +1 but untested I figure lets get it in because it's blocking progress on docker work !

        Please test it good before pushing. Code looks good to me!

        Show
        jayunit100 jay vyas added a comment - - edited +1 but untested I figure lets get it in because it's blocking progress on docker work ! Please test it good before pushing. Code looks good to me!
        Hide
        evans_ye Evans Ye added a comment -

        Thanks, let me test this with newest git repo.

        Show
        evans_ye Evans Ye added a comment - Thanks, let me test this with newest git repo.
        Hide
        evans_ye Evans Ye added a comment -

        I've re-test this on centos and debian, both works well.
        Committed. Thanks for the review jay vyas.

        Show
        evans_ye Evans Ye added a comment - I've re-test this on centos and debian, both works well. Committed. Thanks for the review jay vyas .

          People

          • Assignee:
            evans_ye Evans Ye
            Reporter:
            evans_ye Evans Ye
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development