Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2615

Provide a tool to build pseudo cluster docker images

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: docker
    • Labels:
      None

      Description

      This JIRA implements a tool named Hadoocker, which is to build docker images that have specified hadoop components provisioned inside. During image build time, we execute puppet apply to install packages and configuration. After images are built, components can be brought up quickly by executing puppet apply again. Usually it takes about 80~180 seconds, depends on the complexity of components.
      This work adopt exactly the same idea introduced by Konstantin Boudnik in BIGTOP-2296, but goes further to implement a tool that supports different OS, components, or even a customized site.yaml config file.

      Use cases for Bigtop users:

      • Quick POC environment
      • Being leveraged in CI/CD pipeline

      Use cases for Bigtop:

      • Image generation CI jobs can be treated as our integration tests.

        Issue Links

          Activity

          Hide
          evans_ye Evans Ye added a comment -

          Understood. Let me try to polish the tool by enabling your use case.

          Show
          evans_ye Evans Ye added a comment - Understood. Let me try to polish the tool by enabling your use case.
          Hide
          warwithin YoungWoo Kim added a comment -

          Evans Ye The BOM for Bigtop in our internal repository is customized one – pulling sources from our internal repository, additional components and etc. Sometimes production repository does not reflect our dev environment. So I would like to build the 'sandbox' from my end. Simply I want to pull the packages from './gradlew yum' rather than yum repository via http or ftp.

          Show
          warwithin YoungWoo Kim added a comment - Evans Ye The BOM for Bigtop in our internal repository is customized one – pulling sources from our internal repository, additional components and etc. Sometimes production repository does not reflect our dev environment. So I would like to build the 'sandbox' from my end. Simply I want to pull the packages from './gradlew yum' rather than yum repository via http or ftp.
          Hide
          evans_ye Evans Ye added a comment -

          There is no direct support but you can set REPO=file:///... in sandbox-env.sh. I should make some configureable variable in that file and comment them out.
          What"s the use case you have?
          Any suggestion is appreciated. Thanks.

          Show
          evans_ye Evans Ye added a comment - There is no direct support but you can set REPO= file:/// ... in sandbox-env.sh. I should make some configureable variable in that file and comment them out. What"s the use case you have? Any suggestion is appreciated. Thanks.
          Hide
          warwithin YoungWoo Kim added a comment -

          Evans Ye Just curious... Is it possible to use local repo? I'd like to pull the packages from $BIGTOP_HOME/output/

          Show
          warwithin YoungWoo Kim added a comment - Evans Ye Just curious... Is it possible to use local repo? I'd like to pull the packages from $BIGTOP_HOME/output/
          Hide
          evans_ye Evans Ye added a comment -

          Committed.
          Once we have 1.2 preview repos, let me update the scripts here.
          Thanks for the review.

          Show
          evans_ye Evans Ye added a comment - Committed. Once we have 1.2 preview repos, let me update the scripts here. Thanks for the review.
          Hide
          warwithin YoungWoo Kim added a comment -

          Evans Ye Nice work! IMHO, It's good to go. Later, we can refine this further.

          Show
          warwithin YoungWoo Kim added a comment - Evans Ye Nice work! IMHO, It's good to go. Later, we can refine this further.
          Hide
          evans_ye Evans Ye added a comment -

          Hi all, I'd like to get this merged as an experimental feature in 1.2.
          The feature can be verified via the ci build:
          https://ci.bigtop.apache.org/view/Docker/job/Docker-Sandbox/
          If no objection, I'll get it in.

          Show
          evans_ye Evans Ye added a comment - Hi all, I'd like to get this merged as an experimental feature in 1.2. The feature can be verified via the ci build: https://ci.bigtop.apache.org/view/Docker/job/Docker-Sandbox/ If no objection, I'll get it in.
          Hide
          evans_ye Evans Ye added a comment -

          Renamed the tool to Bigtop Sandbox as this is much easier for user to understand.

          Show
          evans_ye Evans Ye added a comment - Renamed the tool to Bigtop Sandbox as this is much easier for user to understand.
          Hide
          evans_ye Evans Ye added a comment - - edited

          Currently systemd based deployment are not supported because there's no way to get CAP_SYS_ADMIN during image build time. I've noted that in README.md.

          OTOH, I've drafted a jenkins job for producing Hadoocker images:
          https://ci.bigtop.apache.org/view/Docker/job/Hadoocker/

          And pushed one testing image for test:

          docker run -ti --name mystack --privileged --rm -p 8088:8088 bigtop/hadoocker:centos-6_hadoop_yarn_spark
          docker exec -ti mystack /usr/lib/spark/bin/spark-shell --master yarn
          
          Show
          evans_ye Evans Ye added a comment - - edited Currently systemd based deployment are not supported because there's no way to get CAP_SYS_ADMIN during image build time. I've noted that in README.md. OTOH, I've drafted a jenkins job for producing Hadoocker images: https://ci.bigtop.apache.org/view/Docker/job/Hadoocker/ And pushed one testing image for test: docker run -ti --name mystack --privileged --rm -p 8088:8088 bigtop/hadoocker:centos-6_hadoop_yarn_spark docker exec -ti mystack /usr/lib/spark/bin/spark-shell --master yarn
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user evans-ye opened a pull request:

          https://github.com/apache/bigtop/pull/165

          BIGTOP-2615. Provide a tool to build pseudo cluster docker images

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/evans-ye/bigtop BIGTOP-2615

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/165.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #165


          commit 1d6c547054879e3fa34aa32d74b303782f059832
          Author: Evans Ye <evansye@apache.org>
          Date: 2016-12-02T15:01:50Z

          BIGTOP-2615. Provide a tool to build pseudo cluster docker images


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user evans-ye opened a pull request: https://github.com/apache/bigtop/pull/165 BIGTOP-2615 . Provide a tool to build pseudo cluster docker images You can merge this pull request into a Git repository by running: $ git pull https://github.com/evans-ye/bigtop BIGTOP-2615 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/165.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #165 commit 1d6c547054879e3fa34aa32d74b303782f059832 Author: Evans Ye <evansye@apache.org> Date: 2016-12-02T15:01:50Z BIGTOP-2615 . Provide a tool to build pseudo cluster docker images
          Hide
          evans_ye Evans Ye added a comment -

          Design concept attached.

          Show
          evans_ye Evans Ye added a comment - Design concept attached.

            People

            • Assignee:
              evans_ye Evans Ye
              Reporter:
              evans_ye Evans Ye
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development