Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1517

refactor vagrant provisioners to be configurable by yaml file

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 1.0.0
    • Component/s: deployment, vm
    • Labels:
      None

      Description

      In order to update the configurations of our Bigtop Hadoop provisioners(docker-puppet and vagrant-puppet), we need to modify its Vagrantfile directly. It would be better to have a yaml file for configuration management which consolidate all the things we typically care about when provisioning a cluster.

        Activity

        Hide
        jayunit100 jay vyas added a comment -

        thanks evans ! commited.
        now we can easily reconfigure our vagrant recipes to be n nodes or m memory and so on.

        • fyi we will need to do comprehensive tests now of docker/vbox/so on recipes to make sure it all works w latest upgraded failure tests smoke tests and so on soon.
        Show
        jayunit100 jay vyas added a comment - thanks evans ! commited. now we can easily reconfigure our vagrant recipes to be n nodes or m memory and so on. fyi we will need to do comprehensive tests now of docker/vbox/so on recipes to make sure it all works w latest upgraded failure tests smoke tests and so on soon.
        Hide
        evans_ye Evans Ye added a comment -

        Okay jay vyas, I think I'll be ok with this.
        Besides that, both provisioned vagrant and docker cluster are function normally, so I think It should be ok to have this integrated in our code line.

        Show
        evans_ye Evans Ye added a comment - Okay jay vyas , I think I'll be ok with this. Besides that, both provisioned vagrant and docker cluster are function normally, so I think It should be ok to have this integrated in our code line.
        Hide
        jayunit100 jay vyas added a comment - - edited

        hi Evans Ye . im reviewing this now. it looks like it works on vbox, i just ran it.

        the docker failures ? I wouldnt worry, looks like your tests ran just fine, but a minor break (11 minutes 39 seconds mean several mapred tests ran).

        Ill wait for your feedback before final +1/-1, but to me, this looks like a great improvement

        Show
        jayunit100 jay vyas added a comment - - edited hi Evans Ye . im reviewing this now. it looks like it works on vbox, i just ran it. the docker failures ? I wouldnt worry, looks like your tests ran just fine, but a minor break (11 minutes 39 seconds mean several mapred tests ran). Ill wait for your feedback before final +1/-1, but to me, this looks like a great improvement
        Hide
        evans_ye Evans Ye added a comment -

        The patch is ready now!
        I've tested it on virtualbox and docker provisioner, both work fine except running the smoke-tests on docker.
        I got following error during the test:

        :mapreduce:test (Thread[main,5,main]) started.
        :mapreduce:test
        Executing task ':mapreduce:test' (up-to-date check took 1.006 secs) due to:
          No history is available.
        ENV VARIABLE: HADOOP_CONF_DIR = /etc/hadoop/conf/
        [ant:null] Building jar: /root/.gradle/caches/2.1/workerMain/gradle-worker.jar
        Starting process 'Gradle Test Executor 2'. Working directory: /bigtop-home/bigtop-tests/s                              moke-tests/mapreduce Command: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71.x86_64/bin/java -D                              java.security.manager=jarjar.org.gradle.process.internal.child.BootstrapSecurityManager -                              Dfile.encoding=US-ASCII -Duser.country=US -Duser.language=en -Duser.variant -ea -cp /root                              /.gradle/caches/2.1/workerMain/gradle-worker.jar jarjar.org.gradle.process.internal.launc                              her.GradleWorkerMain 'Gradle Test Executor 2'
        Successfully started process 'Gradle Test Executor 2'
        Gradle Test Executor 2 started executing tests.
        
        org.apache.bigtop.itest.hadoop.mapreduce.TestHadoopExamples STANDARD_ERROR
            log4j:WARN No appenders could be found for logger (java.lang.Object).
            log4j:WARN Please initialize the log4j system properly.
            log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
        mapreduce:test FAILED
        :mapreduce:test (Thread[main,5,main]) completed. Took 11 mins 39.408 secs.
        
        FAILURE: Build failed with an exception.
        
        * What went wrong:
        Execution failed for task ':mapreduce:test'.
        > Process 'Gradle Test Executor 2' finished with non-zero exit value 137
        
        * Try:
        Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output.
        
        BUILD FAILED
        
        Total time: 14 mins 13.849 secs
        
        FAILURE: Build failed with an exception.
        
        * What went wrong:
        Process 'Gradle Compiler Daemon 1' finished with non-zero exit value 137
        
        * Try:
        Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output.
        

        I haven't seen that before, so I cancel the patch and run docker provisioner again, surprisingly, I got the same error.
        The pattern looks like the same as the comment posted by jay in BIGTOP-1524.
        So I'm wondering if this is an expected result, jay vyas?

        Show
        evans_ye Evans Ye added a comment - The patch is ready now! I've tested it on virtualbox and docker provisioner, both work fine except running the smoke-tests on docker. I got following error during the test: :mapreduce:test ( Thread [main,5,main]) started. :mapreduce:test Executing task ':mapreduce:test' (up-to-date check took 1.006 secs) due to: No history is available. ENV VARIABLE: HADOOP_CONF_DIR = /etc/hadoop/conf/ [ant: null ] Building jar: /root/.gradle/caches/2.1/workerMain/gradle-worker.jar Starting process 'Gradle Test Executor 2'. Working directory: /bigtop-home/bigtop-tests/s moke-tests/mapreduce Command: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71.x86_64/bin/java -D java.security.manager=jarjar.org.gradle.process.internal.child.BootstrapSecurityManager - Dfile.encoding=US-ASCII -Duser.country=US -Duser.language=en -Duser.variant -ea -cp /root /.gradle/caches/2.1/workerMain/gradle-worker.jar jarjar.org.gradle.process.internal.launc her.GradleWorkerMain 'Gradle Test Executor 2' Successfully started process 'Gradle Test Executor 2' Gradle Test Executor 2 started executing tests. org.apache.bigtop.itest.hadoop.mapreduce.TestHadoopExamples STANDARD_ERROR log4j:WARN No appenders could be found for logger (java.lang. Object ). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http: //logging.apache.org/log4j/1.2/faq.html#noconfig for more info. mapreduce:test FAILED :mapreduce:test ( Thread [main,5,main]) completed. Took 11 mins 39.408 secs. FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':mapreduce:test'. > Process 'Gradle Test Executor 2' finished with non-zero exit value 137 * Try: Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. BUILD FAILED Total time: 14 mins 13.849 secs FAILURE: Build failed with an exception. * What went wrong: Process 'Gradle Compiler Daemon 1' finished with non-zero exit value 137 * Try: Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. I haven't seen that before, so I cancel the patch and run docker provisioner again, surprisingly, I got the same error. The pattern looks like the same as the comment posted by jay in BIGTOP-1524 . So I'm wondering if this is an expected result, jay vyas ?
        Hide
        jayunit100 jay vyas added a comment - - edited

        Hi cos ...... YAML is just like json. It is native to ruby, so in Vagrantfile you have:

        require 'yaml'
        @cluster = YAML.load_file('cluster.yml')
        puts @cluster.inspect
        ## Read in by Vagrantfile...
        TOTAL=@cluster['TOTAL']
        

        meanwhile cluster.yml looks like this :

        TOTAL: 3
        ip:
        OFFSET: 10
        PREFIX: '172.31.16'
        HOST: 'ip-172-31-16-'
        NODE:
        - default: 'm1.small'
        MASTER: 'master.rhbd'
        SUBNET: 'subnet-ad2d2ceb'
        AMI: 'ami-759dcb74'
        REGION: 'ap-northeast-1'
        SERVERS: !!omap
        master:
        type: "m1.small"
        slave:
        type: "m1.small"
        masters:
        - 0
        slaves: #slaves nodes i->j
        - 1
        - 2
        - 3 
        

        So the user just updates the YAML file. We could easily do JSON as well. YAML is used in puppet as well i think quite often (hiera).

        Show
        jayunit100 jay vyas added a comment - - edited Hi cos ...... YAML is just like json. It is native to ruby, so in Vagrantfile you have: require 'yaml' @cluster = YAML.load_file('cluster.yml') puts @cluster.inspect ## Read in by Vagrantfile... TOTAL=@cluster['TOTAL'] meanwhile cluster.yml looks like this : TOTAL: 3 ip: OFFSET: 10 PREFIX: '172.31.16' HOST: 'ip-172-31-16-' NODE: - default: 'm1.small' MASTER: 'master.rhbd' SUBNET: 'subnet-ad2d2ceb' AMI: 'ami-759dcb74' REGION: 'ap-northeast-1' SERVERS: !!omap master: type: "m1.small" slave: type: "m1.small" masters: - 0 slaves: #slaves nodes i->j - 1 - 2 - 3 So the user just updates the YAML file. We could easily do JSON as well. YAML is used in puppet as well i think quite often (hiera).
        Hide
        cos Konstantin Boudnik added a comment -

        Is YAML specific for Docker or Vigrant? Or this'd be just an aesthetic preference?

        Show
        cos Konstantin Boudnik added a comment - Is YAML specific for Docker or Vigrant? Or this'd be just an aesthetic preference?
        Hide
        jayunit100 jay vyas added a comment -

        okay, well , sounds good to me !

        Show
        jayunit100 jay vyas added a comment - okay, well , sounds good to me !
        Hide
        evans_ye Evans Ye added a comment - - edited

        I planed to have separated yaml file for each provisioner because of following reasons:

        • Docker provisioner is slightly different than virtualbox one, for example, we can also configure boot2docker vm in docker's yaml, while these configurations do not need to be added in virtualbox.
        • By separating yaml file, users can launch 2 type of cluster simultaneously for different purpose.
        Show
        evans_ye Evans Ye added a comment - - edited I planed to have separated yaml file for each provisioner because of following reasons: Docker provisioner is slightly different than virtualbox one, for example, we can also configure boot2docker vm in docker's yaml, while these configurations do not need to be added in virtualbox. By separating yaml file, users can launch 2 type of cluster simultaneously for different purpose.
        Hide
        jayunit100 jay vyas added a comment -

        Thanks evans. Should the YAML file contain docker as well as Vagrant specific settings, or separate files for each provider?

        Show
        jayunit100 jay vyas added a comment - Thanks evans. Should the YAML file contain docker as well as Vagrant specific settings, or separate files for each provider?

          People

          • Assignee:
            evans_ye Evans Ye
            Reporter:
            evans_ye Evans Ye
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development