Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: backlog
    • Fix Version/s: 1.2.0
    • Component/s: docker, provisioner
    • Labels:
      None

      Issue Links

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user saulshanabrook commented on the issue:

        https://github.com/apache/bigtop/pull/74

        Would it be helpful for me to create a new PR with these conflicts resolved?

        Show
        githubbot ASF GitHub Bot added a comment - Github user saulshanabrook commented on the issue: https://github.com/apache/bigtop/pull/74 Would it be helpful for me to create a new PR with these conflicts resolved?
        Hide
        oflebbe Olaf Flebbe added a comment -

        pushed by merge

        Show
        oflebbe Olaf Flebbe added a comment - pushed by merge
        Hide
        evans_ye Evans Ye added a comment -

        I don't even know which PR as well. But I'm sure the newest commit is here:
        https://github.com/apache/bigtop/commit/d19b5c70a30d9b4e3c1607dd01cb967ad9c0c30b
        There're no much issue I can recall, but just need to redesign the system better. For instance, the port forwarding config in our yaml should be removed. I'll resume work on this and add the swarm support.

        Show
        evans_ye Evans Ye added a comment - I don't even know which PR as well. But I'm sure the newest commit is here: https://github.com/apache/bigtop/commit/d19b5c70a30d9b4e3c1607dd01cb967ad9c0c30b There're no much issue I can recall, but just need to redesign the system better. For instance, the port forwarding config in our yaml should be removed. I'll resume work on this and add the swarm support.
        Hide
        oflebbe Olaf Flebbe added a comment -

        Hi Evans Ye : Where do I find your lastet patch at the github pull request or attached at this JIRA ?

        Are there problems left ?

        Show
        oflebbe Olaf Flebbe added a comment - Hi Evans Ye : Where do I find your lastet patch at the github pull request or attached at this JIRA ? Are there problems left ?
        Hide
        cos Konstantin Boudnik added a comment -

        One other thing: because this is going to be a new top-level directory, let's make sure it is added to
        src/assembly/release-assembly.xml for production of the release source artifact.

        Show
        cos Konstantin Boudnik added a comment - One other thing: because this is going to be a new top-level directory, let's make sure it is added to src/assembly/release-assembly.xml for production of the release source artifact.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user evans-ye commented on the pull request:

        https://github.com/apache/bigtop/pull/74#issuecomment-174254879

        Sure. That what I plan to do in BIGTOP-2273.

        Show
        githubbot ASF GitHub Bot added a comment - Github user evans-ye commented on the pull request: https://github.com/apache/bigtop/pull/74#issuecomment-174254879 Sure. That what I plan to do in BIGTOP-2273 .
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user c0s commented on the pull request:

        https://github.com/apache/bigtop/pull/74#issuecomment-174245170

        Sorry, you're right. Getting tired...
        Do you think we need to provide a mechanism for ppl to get into these containers via ssh or 'docker attach' or else? Thanks!

        Show
        githubbot ASF GitHub Bot added a comment - Github user c0s commented on the pull request: https://github.com/apache/bigtop/pull/74#issuecomment-174245170 Sorry, you're right. Getting tired... Do you think we need to provide a mechanism for ppl to get into these containers via ssh or 'docker attach' or else? Thanks!
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user evans-ye commented on the pull request:

        https://github.com/apache/bigtop/pull/74#issuecomment-174244307

        It's in the right place:
        https://github.com/apache/bigtop/blob/BIGTOP-2253/provisioner/docker/README.md
        Would you mind to check again?

        Show
        githubbot ASF GitHub Bot added a comment - Github user evans-ye commented on the pull request: https://github.com/apache/bigtop/pull/74#issuecomment-174244307 It's in the right place: https://github.com/apache/bigtop/blob/BIGTOP-2253/provisioner/docker/README.md Would you mind to check again?
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user c0s commented on the pull request:

        https://github.com/apache/bigtop/pull/74#issuecomment-174243946

        I don't see bigtop/provisioner/docker/README.md file at all. Looks like that changes to README were made to bigtop-deploy/vm/vagrant-provisioner-docker/REAME.md
        Sorry if I am confusing something, but the first location doesn't seem to have any readme files.

        Show
        githubbot ASF GitHub Bot added a comment - Github user c0s commented on the pull request: https://github.com/apache/bigtop/pull/74#issuecomment-174243946 I don't see bigtop/provisioner/docker/README.md file at all. Looks like that changes to README were made to bigtop-deploy/vm/vagrant-provisioner-docker/REAME.md Sorry if I am confusing something, but the first location doesn't seem to have any readme files.
        Hide
        evans_ye Evans Ye added a comment -

        Yes. You're right!

        Show
        evans_ye Evans Ye added a comment - Yes. You're right!
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user evans-ye commented on the pull request:

        https://github.com/apache/bigtop/pull/74#issuecomment-174243576

        I'm not quite sure what you mean by:
        It is a bit awkward to read about compose in the README belonging to vagrant provisioner. No?

        Right now we have bigtop/provisioner/docker/README.md for docker-compose.
        There will be bigtop/provisioner/vagrant/REAME.md for vagrant provisioner onece we moved it from bigotp-deploy.

        Show
        githubbot ASF GitHub Bot added a comment - Github user evans-ye commented on the pull request: https://github.com/apache/bigtop/pull/74#issuecomment-174243576 I'm not quite sure what you mean by: It is a bit awkward to read about compose in the README belonging to vagrant provisioner. No? Right now we have bigtop/provisioner/docker/README.md for docker-compose. There will be bigtop/provisioner/vagrant/REAME.md for vagrant provisioner onece we moved it from bigotp-deploy.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user c0s commented on the pull request:

        https://github.com/apache/bigtop/pull/74#issuecomment-174243289

        One more though: perhaps, it makes sense to add a version of README to the new provisioner directory? It is a bit awkward to read about compose in the README belonging to vagrant provisioner. No?

        Show
        githubbot ASF GitHub Bot added a comment - Github user c0s commented on the pull request: https://github.com/apache/bigtop/pull/74#issuecomment-174243289 One more though: perhaps, it makes sense to add a version of README to the new provisioner directory? It is a bit awkward to read about compose in the README belonging to vagrant provisioner. No?
        Hide
        cos Konstantin Boudnik added a comment -

        Let's keep it open, until we are done merging this into the mainline. Otherwise, it might slip through the cracks.

        Show
        cos Konstantin Boudnik added a comment - Let's keep it open, until we are done merging this into the mainline. Otherwise, it might slip through the cracks.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user evans-ye commented on the pull request:

        https://github.com/apache/bigtop/pull/74#issuecomment-174241978

        Awesome! Glad it works!

        Show
        githubbot ASF GitHub Bot added a comment - Github user evans-ye commented on the pull request: https://github.com/apache/bigtop/pull/74#issuecomment-174241978 Awesome! Glad it works!
        Hide
        evans_ye Evans Ye added a comment -

        Checked in to BIGTOP-2253 branch. Plan to be merged in 1.2 release.

        Show
        evans_ye Evans Ye added a comment - Checked in to BIGTOP-2253 branch. Plan to be merged in 1.2 release.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user c0s commented on the pull request:

        https://github.com/apache/bigtop/pull/74#issuecomment-174241420

        Yeah, looks like I know what my problem with the installation was. I've tried to do 'pip install' which was an idiotic idea, cause nothing really works in Python world out of the box. the way you suggested it, indeed, worked well. Thanks!

        Show
        githubbot ASF GitHub Bot added a comment - Github user c0s commented on the pull request: https://github.com/apache/bigtop/pull/74#issuecomment-174241420 Yeah, looks like I know what my problem with the installation was. I've tried to do 'pip install' which was an idiotic idea, cause nothing really works in Python world out of the box. the way you suggested it, indeed, worked well. Thanks!
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user c0s commented on the pull request:

        https://github.com/apache/bigtop/pull/74#issuecomment-174241255

        Cool, and I will use of the branch to test in my environment. Thanks!

        Show
        githubbot ASF GitHub Bot added a comment - Github user c0s commented on the pull request: https://github.com/apache/bigtop/pull/74#issuecomment-174241255 Cool, and I will use of the branch to test in my environment. Thanks!
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user evans-ye commented on the pull request:

        https://github.com/apache/bigtop/pull/74#issuecomment-174241247

        I'll check in to a branch and iterate on it. That way I can also setup a CI for it. Thanks for the review!

        Show
        githubbot ASF GitHub Bot added a comment - Github user evans-ye commented on the pull request: https://github.com/apache/bigtop/pull/74#issuecomment-174241247 I'll check in to a branch and iterate on it. That way I can also setup a CI for it. Thanks for the review!
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user evans-ye commented on the pull request:

        https://github.com/apache/bigtop/pull/74#issuecomment-174241188

        I just tried. It should work instantly

        Show
        githubbot ASF GitHub Bot added a comment - Github user evans-ye commented on the pull request: https://github.com/apache/bigtop/pull/74#issuecomment-174241188 I just tried. It should work instantly
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user c0s commented on the pull request:

        https://github.com/apache/bigtop/pull/74#issuecomment-174241170

        Dunno, I've tried it about a month ago - it was a complete mess. I will try again today or tomorrow and comment here. Thanks!

        Show
        githubbot ASF GitHub Bot added a comment - Github user c0s commented on the pull request: https://github.com/apache/bigtop/pull/74#issuecomment-174241170 Dunno, I've tried it about a month ago - it was a complete mess. I will try again today or tomorrow and comment here. Thanks!
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user evans-ye commented on the pull request:

        https://github.com/apache/bigtop/pull/74#issuecomment-174241114

        How so? It should be just two command:
        curl L https://github.com/docker/compose/releases/download/1.5.2/docker-compose-`uname -s``uname -m` > /usr/local/bin/docker-compose
        chmod +x /usr/local/bin/docker-compose

        The docker engine, OTOH, is much complicated on ubuntu 14...

        Show
        githubbot ASF GitHub Bot added a comment - Github user evans-ye commented on the pull request: https://github.com/apache/bigtop/pull/74#issuecomment-174241114 How so? It should be just two command: curl L https://github.com/docker/compose/releases/download/1.5.2/docker-compose- `uname -s` `uname -m` > /usr/local/bin/docker-compose chmod +x /usr/local/bin/docker-compose The docker engine, OTOH, is much complicated on ubuntu 14...
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user c0s commented on the pull request:

        https://github.com/apache/bigtop/pull/74#issuecomment-174240787

        Let's do the branch and keep on working around it. I think your stuff is pretty much ready, but want to play with it a little bit if you don't mind.

        Show
        githubbot ASF GitHub Bot added a comment - Github user c0s commented on the pull request: https://github.com/apache/bigtop/pull/74#issuecomment-174240787 Let's do the branch and keep on working around it. I think your stuff is pretty much ready, but want to play with it a little bit if you don't mind.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user evans-ye commented on the pull request:

        https://github.com/apache/bigtop/pull/74#issuecomment-174240676

        No. I'll still go to a feature branch so that we don't have confusing things inside 1.1 release.
        The feature is completely switched from vagrant to docker-compose. I've removed vagrant name entirely in the new commit.

        Show
        githubbot ASF GitHub Bot added a comment - Github user evans-ye commented on the pull request: https://github.com/apache/bigtop/pull/74#issuecomment-174240676 No. I'll still go to a feature branch so that we don't have confusing things inside 1.1 release. The feature is completely switched from vagrant to docker-compose. I've removed vagrant name entirely in the new commit.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user evans-ye commented on the pull request:

        https://github.com/apache/bigtop/pull/74#issuecomment-174240159

        OK. I've pushed a new commit:
        https://github.com/evans-ye/bigtop/commit/b6579c923576fa748e8c70ea6639e3348cb4ec41

        Basically updated the REAME to drop vagrant and switch to docker-compose.
        I've also rename the config file name all around the code.

        This is for you to review what has been changed.
        I'll strip these two into one patch when actually committed.
        Thanks!

        Show
        githubbot ASF GitHub Bot added a comment - Github user evans-ye commented on the pull request: https://github.com/apache/bigtop/pull/74#issuecomment-174240159 OK. I've pushed a new commit: https://github.com/evans-ye/bigtop/commit/b6579c923576fa748e8c70ea6639e3348cb4ec41 Basically updated the REAME to drop vagrant and switch to docker-compose. I've also rename the config file name all around the code. This is for you to review what has been changed. I'll strip these two into one patch when actually committed. Thanks!
        Hide
        evans_ye Evans Ye added a comment -

        Create a new PR to put Bigtop Docker Provisioner under bigtop/provisioner/docker. The vagrant-puppet-docker is unchanged and still binding to the gradle task: docker-provisioner.

        Show
        evans_ye Evans Ye added a comment - Create a new PR to put Bigtop Docker Provisioner under bigtop/provisioner/docker . The vagrant-puppet-docker is unchanged and still binding to the gradle task: docker-provisioner.
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user evans-ye opened a pull request:

        https://github.com/apache/bigtop/pull/74

        BIGTOP-2254. Replace the Docker orchestration tool from Vagrant to Do…

        …cker Compose

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/evans-ye/bigtop BIGTOP-2254

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/bigtop/pull/74.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #74


        commit 23f56d0351d1ed8d92d0923ecfd172bcac60dca5
        Author: Evans Ye <evansye@apache.org>
        Date: 2016-01-23T10:33:03Z

        BIGTOP-2254. Replace the Docker orchestration tool from Vagrant to Docker Compose


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user evans-ye opened a pull request: https://github.com/apache/bigtop/pull/74 BIGTOP-2254 . Replace the Docker orchestration tool from Vagrant to Do… …cker Compose You can merge this pull request into a Git repository by running: $ git pull https://github.com/evans-ye/bigtop BIGTOP-2254 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/74.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #74 commit 23f56d0351d1ed8d92d0923ecfd172bcac60dca5 Author: Evans Ye <evansye@apache.org> Date: 2016-01-23T10:33:03Z BIGTOP-2254 . Replace the Docker orchestration tool from Vagrant to Docker Compose
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user evans-ye closed the pull request at:

        https://github.com/apache/bigtop/pull/71

        Show
        githubbot ASF GitHub Bot added a comment - Github user evans-ye closed the pull request at: https://github.com/apache/bigtop/pull/71
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user evans-ye commented on the pull request:

        https://github.com/apache/bigtop/pull/71#issuecomment-173178445

        Thanks for the review!

        Yes you should have docker-compose installed first. I plan to add support to download and install it via docker-hadoop as well. After all it just need one line to get it done.

        config/hieradata will be generated in generate-config function. In the future we should allow users to specify customized config instead of always generate new ones.

        The new directory is a very good idea. I was digging to deep so that I can't see the name is so misleading... With that we can just merge it into master w/o breaking anything.

        In fact, I'm thinking that maybe we can put a top level directory for provisioners. That things currently sit inside bigtop-deploy is hard to find. What do you think?

        BTW, any shell script styling comment is welcome.

        Show
        githubbot ASF GitHub Bot added a comment - Github user evans-ye commented on the pull request: https://github.com/apache/bigtop/pull/71#issuecomment-173178445 Thanks for the review! Yes you should have docker-compose installed first. I plan to add support to download and install it via docker-hadoop as well. After all it just need one line to get it done. config/hieradata will be generated in generate-config function. In the future we should allow users to specify customized config instead of always generate new ones. The new directory is a very good idea. I was digging to deep so that I can't see the name is so misleading... With that we can just merge it into master w/o breaking anything. In fact, I'm thinking that maybe we can put a top level directory for provisioners. That things currently sit inside bigtop-deploy is hard to find. What do you think? BTW, any shell script styling comment is welcome.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user c0s commented on the pull request:

        https://github.com/apache/bigtop/pull/71#issuecomment-172649679

        I will start looking at it in a day or two as soon as I am done with a couple of things in the woodworks right now.

        Show
        githubbot ASF GitHub Bot added a comment - Github user c0s commented on the pull request: https://github.com/apache/bigtop/pull/71#issuecomment-172649679 I will start looking at it in a day or two as soon as I am done with a couple of things in the woodworks right now.
        Hide
        cos Konstantin Boudnik added a comment -

        Evans, with branches any committer can create them/delete them once not needed. I think your idea makes a lot of sense: let's spin it on the CI/in personal tests to make sure it is all good. This way, other ppl might be able to contribute as well. Thanks for working on it!

        Show
        cos Konstantin Boudnik added a comment - Evans, with branches any committer can create them/delete them once not needed. I think your idea makes a lot of sense: let's spin it on the CI/in personal tests to make sure it is all good. This way, other ppl might be able to contribute as well. Thanks for working on it!
        Hide
        evans_ye Evans Ye added a comment -

        BTW, the gradlew docker-provisioner task is transparent to the change so you should expect everything just works.

        Show
        evans_ye Evans Ye added a comment - BTW, the gradlew docker-provisioner task is transparent to the change so you should expect everything just works.
        Hide
        evans_ye Evans Ye added a comment -

        This is a huge rewrite of the whole Docker Provisioner, I suggest to commit this patch into a separated branch: BIGTOP-2253, and put it on CI for test first.
        Konstantin Boudnik How do you think? Is there a policy for creating branches or I can just commit & push?

        Show
        evans_ye Evans Ye added a comment - This is a huge rewrite of the whole Docker Provisioner, I suggest to commit this patch into a separated branch: BIGTOP-2253 , and put it on CI for test first. Konstantin Boudnik How do you think? Is there a policy for creating branches or I can just commit & push?
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user evans-ye opened a pull request:

        https://github.com/apache/bigtop/pull/71

        BIGTOP-2254. Replace the Docker orchestration tool from Vagrant to Do…

        …cker Compose

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/evans-ye/bigtop BIGTOP-2254

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/bigtop/pull/71.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #71


        commit 89b64afff5b20a3f071b91ea39c60bd266ff5ae7
        Author: Evans Ye <evansye@apache.org>
        Date: 2016-01-17T06:36:35Z

        BIGTOP-2254. Replace the Docker orchestration tool from Vagrant to Docker Compose


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user evans-ye opened a pull request: https://github.com/apache/bigtop/pull/71 BIGTOP-2254 . Replace the Docker orchestration tool from Vagrant to Do… …cker Compose You can merge this pull request into a Git repository by running: $ git pull https://github.com/evans-ye/bigtop BIGTOP-2254 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/71.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #71 commit 89b64afff5b20a3f071b91ea39c60bd266ff5ae7 Author: Evans Ye <evansye@apache.org> Date: 2016-01-17T06:36:35Z BIGTOP-2254 . Replace the Docker orchestration tool from Vagrant to Docker Compose

          People

          • Assignee:
            evans_ye Evans Ye
            Reporter:
            evans_ye Evans Ye
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development