Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: provisioner
    • Labels:
      None

      Description

      Hi Evans Ye,

      I think the brach BIGTOP-2253 can be improved. Hope you like it.

      Olaf

      1. BIGTOP-2505.1.patch
        10 kB
        Olaf Flebbe
      2. BIGTOP-2505.2.patch
        8 kB
        Olaf Flebbe
      3. BIGTOP-2505.3.patch
        10 kB
        Olaf Flebbe
      4. BIGTOP-2505.4.patch
        10 kB
        Olaf Flebbe

        Activity

        Hide
        oflebbe Olaf Flebbe added a comment -

        To apply, first rebase branch to current master

        Show
        oflebbe Olaf Flebbe added a comment - To apply, first rebase branch to current master
        Hide
        oflebbe Olaf Flebbe added a comment -

        I like to apply it to branch

        Show
        oflebbe Olaf Flebbe added a comment - I like to apply it to branch
        Hide
        evans_ye Evans Ye added a comment - - edited

        Hi Olaf Flebbe I'm really happy to see you to show your love to these JIRAs.
        However, I found that the branch can not successfully startup a cluster on Docker for Mac, I've fixed it locally, but haven't made a patch yet.
        Sorry for the delay of merging...

        Anyway, there're something I'd like to discuss with you regarding to this patch:

        • We can't remove bash /bigtop-home/bigtop_toolchain/bin/puppetize.sh because vagrant-puppet-vm is still using it. We can change the box: "puppetlabs/centos-7.0-64-nocm" to be a puppetized box, but we also need to make sure that vagrant-puppet-openstack won't be effected, which I never have an environment to test it yet. . Maybe we just don't touch it this time?
        • I'm not sure /sbin/init is gonna work for all distro. It can't run on centos 6.
        • ../../bigtop-deploy/puppet/hiera.yaml:/etc/puppet/hiera.yaml: I think for some day in the future we can make the images self-contained with code and config included, hence I try to centralize a copy of all configs.
        • echo > ./config/hosts this is pretty.
        • cp -rf $BIGTOP_PUPPET_DIR/hieradata ./config/ would be better to output more logs with -v.
        • puppet apply.. -d can output more logs, which somehow looks like a poor version of progress bar, in my opinion.
        • no=$(($1-1)) I did this for a reason that users see the number started from 1 when using docker-compose scale:
          Creating and starting 20160724012000r227_bigtop_1 ... done
          Creating and starting 20160724012000r227_bigtop_2 ... done
          

          Hence starting at 1 would be much more intuitive and easier to guess.

        Show
        evans_ye Evans Ye added a comment - - edited Hi Olaf Flebbe I'm really happy to see you to show your love to these JIRAs. However, I found that the branch can not successfully startup a cluster on Docker for Mac, I've fixed it locally, but haven't made a patch yet. Sorry for the delay of merging... Anyway, there're something I'd like to discuss with you regarding to this patch: We can't remove bash /bigtop-home/bigtop_toolchain/bin/puppetize.sh because vagrant-puppet-vm is still using it. We can change the box: "puppetlabs/centos-7.0-64-nocm" to be a puppetized box, but we also need to make sure that vagrant-puppet-openstack won't be effected, which I never have an environment to test it yet. . Maybe we just don't touch it this time? I'm not sure /sbin/init is gonna work for all distro. It can't run on centos 6. ../../bigtop-deploy/puppet/hiera.yaml:/etc/puppet/hiera.yaml : I think for some day in the future we can make the images self-contained with code and config included, hence I try to centralize a copy of all configs. echo > ./config/hosts this is pretty. cp -rf $BIGTOP_PUPPET_DIR/hieradata ./config/ would be better to output more logs with -v. puppet apply.. -d can output more logs, which somehow looks like a poor version of progress bar, in my opinion. no=$(($1-1)) I did this for a reason that users see the number started from 1 when using docker-compose scale: Creating and starting 20160724012000r227_bigtop_1 ... done Creating and starting 20160724012000r227_bigtop_2 ... done Hence starting at 1 would be much more intuitive and easier to guess.
        Hide
        oflebbe Olaf Flebbe added a comment -

        Hi Evans Ye

        If you got it fixed to run on docker on mac, wonderful. I debugged it for hours and didn't found the root case why the bind mounts didn't work correctly in latest beta.

        1) Ok, I am leaving the puppetize code in it.
        2) /sbin/init worked on centos-6 the last time I checked.
        3) One of my unsuccessful workarounds for mac. Will remove it again
        6) puppet -d gives way to much output and scrolls error messages out of view. Please please let me remove debug
        7) You may have missed that I didn't changed the semantics. It looked a bit cleaner to me.

        Will rework the patch, please add your changes here

        Show
        oflebbe Olaf Flebbe added a comment - Hi Evans Ye If you got it fixed to run on docker on mac, wonderful. I debugged it for hours and didn't found the root case why the bind mounts didn't work correctly in latest beta. 1) Ok, I am leaving the puppetize code in it. 2) /sbin/init worked on centos-6 the last time I checked. 3) One of my unsuccessful workarounds for mac. Will remove it again 6) puppet -d gives way to much output and scrolls error messages out of view. Please please let me remove debug 7) You may have missed that I didn't changed the semantics. It looked a bit cleaner to me. Will rework the patch, please add your changes here
        Hide
        oflebbe Olaf Flebbe added a comment -

        Reworked patch

        Show
        oflebbe Olaf Flebbe added a comment - Reworked patch
        Hide
        oflebbe Olaf Flebbe added a comment -

        Forgot config_centos7.patch

        Show
        oflebbe Olaf Flebbe added a comment - Forgot config_centos7.patch
        Hide
        evans_ye Evans Ye added a comment -

        Olaf Flebbe thanks for the quick patch!
        I do have fix for docker on mac, but I still can't figure out the root cause. Maybe we can commit yours first and I'll open up another JIRA for that.

        Back to this JIRA,

        • For the /sbin/init change, I'm still getting error like below:
          $  ((c5a36fa...))$ ./docker-hadoop.sh -d -c 1
          Environment check...
          Check docker:
          Docker version 1.11.2, build b9f10c9/1.11.2
          Check docker-compose:
          docker-compose version 1.6.2, build 4d72027
          Check ruby:
          ruby 2.0.0p648 (2015-12-16) [x86_64-linux]
          Creating and starting 1 ... done
          Bigtop Puppet configurations are shared between instances, and can be modified under config/hieradata
          ‘../../bigtop-deploy/puppet/hieradata/site.yaml’ -> ‘./config/hieradata/site.yaml’
          ‘../../bigtop-deploy/puppet/hieradata/bigtop’ -> ‘./config/hieradata/bigtop’
          ‘../../bigtop-deploy/puppet/hieradata/bigtop/ha.yaml’ -> ‘./config/hieradata/bigtop/ha.yaml’
          ‘../../bigtop-deploy/puppet/hieradata/bigtop/cluster.yaml’ -> ‘./config/hieradata/bigtop/cluster.yaml’
          ‘../../bigtop-deploy/puppet/hieradata/bigtop/noha.yaml’ -> ‘./config/hieradata/bigtop/noha.yaml’
          Error response from daemon: Container b84ab6348633791cdd80a354e3e74f3e9936ec8c7b9c7b3099c8b74c152e1a34 is not running
          lstat /home/evans/2253/provisioner/docker/centos: no such file or directory
          Error response from daemon: Container b84ab6348633791cdd80a354e3e74f3e9936ec8c7b9c7b3099c8b74c152e1a34 is not running
          Error response from daemon: Container b84ab6348633791cdd80a354e3e74f3e9936ec8c7b9c7b3099c8b74c152e1a34 is not running
          

          The process is not running foreground:

          $  ((c5a36fa...))$ docker ps -a
          CONTAINER ID        IMAGE                    COMMAND                 CREATED             STATUS                     PORTS               NAMES
          f31c679dabbc        bigtop/deploy:centos-6   "/tini -- /sbin/init"   5 seconds ago       Exited (1) 4 seconds ago                       20160725175312r7487_bigtop_1
          

          I'm running on Amazon linux. To be more specifically, our slave02.

        • The downside of removing -d in puppet apply is that the process hang while something is running, which might lead to not-so-good user experience. I feel that the errors in red are easy to identify. But you're right about the scrolling problem. hmm... how do you think Konstantin Boudnik?
        • The config_centos7.yaml is still using bigtop/deploy:centos-6 image, which shouldn't be.

        Would be nice if you can tell me how you get /sbin/init to work.

        Show
        evans_ye Evans Ye added a comment - Olaf Flebbe thanks for the quick patch! I do have fix for docker on mac, but I still can't figure out the root cause. Maybe we can commit yours first and I'll open up another JIRA for that. Back to this JIRA, For the /sbin/init change, I'm still getting error like below: $ ((c5a36fa...))$ ./docker-hadoop.sh -d -c 1 Environment check... Check docker: Docker version 1.11.2, build b9f10c9/1.11.2 Check docker-compose: docker-compose version 1.6.2, build 4d72027 Check ruby: ruby 2.0.0p648 (2015-12-16) [x86_64-linux] Creating and starting 1 ... done Bigtop Puppet configurations are shared between instances, and can be modified under config/hieradata ‘../../bigtop-deploy/puppet/hieradata/site.yaml’ -> ‘./config/hieradata/site.yaml’ ‘../../bigtop-deploy/puppet/hieradata/bigtop’ -> ‘./config/hieradata/bigtop’ ‘../../bigtop-deploy/puppet/hieradata/bigtop/ha.yaml’ -> ‘./config/hieradata/bigtop/ha.yaml’ ‘../../bigtop-deploy/puppet/hieradata/bigtop/cluster.yaml’ -> ‘./config/hieradata/bigtop/cluster.yaml’ ‘../../bigtop-deploy/puppet/hieradata/bigtop/noha.yaml’ -> ‘./config/hieradata/bigtop/noha.yaml’ Error response from daemon: Container b84ab6348633791cdd80a354e3e74f3e9936ec8c7b9c7b3099c8b74c152e1a34 is not running lstat /home/evans/2253/provisioner/docker/centos: no such file or directory Error response from daemon: Container b84ab6348633791cdd80a354e3e74f3e9936ec8c7b9c7b3099c8b74c152e1a34 is not running Error response from daemon: Container b84ab6348633791cdd80a354e3e74f3e9936ec8c7b9c7b3099c8b74c152e1a34 is not running The process is not running foreground: $ ((c5a36fa...))$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f31c679dabbc bigtop/deploy:centos-6 "/tini -- /sbin/init" 5 seconds ago Exited (1) 4 seconds ago 20160725175312r7487_bigtop_1 I'm running on Amazon linux. To be more specifically, our slave02. The downside of removing -d in puppet apply is that the process hang while something is running, which might lead to not-so-good user experience. I feel that the errors in red are easy to identify. But you're right about the scrolling problem. hmm... how do you think Konstantin Boudnik ? The config_centos7.yaml is still using bigtop/deploy:centos-6 image, which shouldn't be. Would be nice if you can tell me how you get /sbin/init to work.
        Hide
        oflebbe Olaf Flebbe added a comment -

        Evans Ye The patch depends on the BIGTOP-2494 which is committed and new images are already pushed to dockerhub. This enables /sbin/init

        Show
        oflebbe Olaf Flebbe added a comment - Evans Ye The patch depends on the BIGTOP-2494 which is committed and new images are already pushed to dockerhub. This enables /sbin/init
        Hide
        evans_ye Evans Ye added a comment -

        I see. That's just because I'm using old image...
        Thanks for pointing me out.
        The only thing left is config_centos7.yaml pointing to bigtop/deploy:centos-6.
        Would you update the patch?

        Show
        evans_ye Evans Ye added a comment - I see. That's just because I'm using old image... Thanks for pointing me out. The only thing left is config_centos7.yaml pointing to bigtop/deploy:centos-6. Would you update the patch?
        Hide
        oflebbe Olaf Flebbe added a comment -

        Oops!

        Show
        oflebbe Olaf Flebbe added a comment - Oops!
        Hide
        oflebbe Olaf Flebbe added a comment -

        Thanks for reviewing. Will Apply this to branch in a minute

        Show
        oflebbe Olaf Flebbe added a comment - Thanks for reviewing. Will Apply this to branch in a minute
        Hide
        oflebbe Olaf Flebbe added a comment -

        Rebased the branch BIGTOP-2253 to current master and applied patch to branch.

        Show
        oflebbe Olaf Flebbe added a comment - Rebased the branch BIGTOP-2253 to current master and applied patch to branch.
        Hide
        oflebbe Olaf Flebbe added a comment -

        thanx

        Show
        oflebbe Olaf Flebbe added a comment - thanx
        Hide
        evans_ye Evans Ye added a comment -

        Hmm, it seem /sbin/init will change the hostname setting so that the following up provision will fail. Still need to identify the root cause...

        Show
        evans_ye Evans Ye added a comment - Hmm, it seem /sbin/init will change the hostname setting so that the following up provision will fail. Still need to identify the root cause...
        Hide
        oflebbe Olaf Flebbe added a comment -

        merged

        Show
        oflebbe Olaf Flebbe added a comment - merged

          People

          • Assignee:
            oflebbe Olaf Flebbe
            Reporter:
            oflebbe Olaf Flebbe
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development