Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5635

Improve Docker tooling to make it easier to build images and launch Flink via Docker tools

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.3.0
    • Component/s: Docker
    • Labels:
      None

      Description

      This is a bit of a catch-all ticket for general improvements to the Flink on Docker experience.

      Things to improve:

      • Make it possible to build a Docker image from your own flink-dist directory as well as official releases.
      • Make it possible to override the image name so a user can more easily publish these images to their Docker repository
      • Provide scripts that show how to properly run on Docker Swarm or similar environments with overlay networking (Kubernetes) without using host networking.
      • Log to stdout rather than to files.
      • Work properly with docker-compose for local deployment as well as production deployments (Swarm/k8s)

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user jgrier opened a pull request:

          https://github.com/apache/flink/pull/3205

          FLINK-5635 Improve Docker tooling to make it easier to build images and launch Flink via Docker tools

          Improvements for Docker on Flink experience.

          This PR depends on this other one: https://github.com/apache/flink/pull/3204

          • Modifying Dockerfile to allow building from local flink-dist as well as release URLs
          • Making image name configurable so it's easier for user's to use this to build an publish their own images
          • Logging to stdout rather than just files
          • Adding scripts to deploy seamlessly on Docker Swarm without host networking
          • Updating Docker Compose scripts to work correctly for deploying locally
          • Generally parameterizing things so these Docker scripts are more generally useful and self-documenting
          • Provides options so that you can deploy multiple Flink services with unique names on Docker Swarm
          • This should all work well with the new Docker "stack" stuff as well.

          Example usage:

          ```

          1. Build a Docker image from your local flink build
            ./build.sh --from-local-dist --image-name "dataartisans/flink"
            ```

          ```

          1. Build a Docker image from an official release
            ./build.sh --from-release --flink-verison 1.2.0 --hadoop-version 2.7 --scala-version 2.11 --image-name "dataartisans/flink"
            ```
            ```
          2. Run with Docker Compose
            docker-compose up -d
            docker-compose scale taskmanager=4
            ```
            ```
          3. Run on Docker Swarm
            ./create-docker-swarm-service.sh --image-name "dataartisans/flink" my-flink-service my-flink-port
            ```
            ```
          4. Shut down
            ./remove-docker-swarm-service.sh --image-name "dataartisans/flink" my-flink-service
            ```

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/jgrier/flink release-1.2

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3205.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3205


          commit d380e98b397c770ee4b7eb9e4abb2387e7672d4c
          Author: Jamie Grier <jamie.grier@gmail.com>
          Date: 2017-01-21T00:12:17Z

          Improvements for Docker on Flink experience.

          Modifying Dockerfile to build from local flink-dist as well as release URLs.
          Logging to stdout.
          Adding scripts to deploy seamlessly on Docker Swarm.
          Updating Docker Compose scripts to work correctly.
          Parameterizing things so these Docker scripts are more generally useful.

          commit 962290ab66d993cac4c4e233a017feb8a6eb1cc5
          Author: Jamie Grier <jamie.grier@gmail.com>
          Date: 2017-01-21T00:14:27Z

          Adding flag to flink-daemon.sh and related scripts so that we can actually log to the console – which is better for Docker environments.

          commit 66c2133ede0ed9914baba2fe7b93f1b9d585ede2
          Author: Jamie Grier <jamie.grier@gmail.com>
          Date: 2017-01-25T05:45:23Z

          Adding Apache license to new files.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user jgrier opened a pull request: https://github.com/apache/flink/pull/3205 FLINK-5635 Improve Docker tooling to make it easier to build images and launch Flink via Docker tools Improvements for Docker on Flink experience. This PR depends on this other one: https://github.com/apache/flink/pull/3204 Modifying Dockerfile to allow building from local flink-dist as well as release URLs Making image name configurable so it's easier for user's to use this to build an publish their own images Logging to stdout rather than just files Adding scripts to deploy seamlessly on Docker Swarm without host networking Updating Docker Compose scripts to work correctly for deploying locally Generally parameterizing things so these Docker scripts are more generally useful and self-documenting Provides options so that you can deploy multiple Flink services with unique names on Docker Swarm This should all work well with the new Docker "stack" stuff as well. Example usage: ``` Build a Docker image from your local flink build ./build.sh --from-local-dist --image-name "dataartisans/flink" ``` ``` Build a Docker image from an official release ./build.sh --from-release --flink-verison 1.2.0 --hadoop-version 2.7 --scala-version 2.11 --image-name "dataartisans/flink" ``` ``` Run with Docker Compose docker-compose up -d docker-compose scale taskmanager=4 ``` ``` Run on Docker Swarm ./create-docker-swarm-service.sh --image-name "dataartisans/flink" my-flink-service my-flink-port ``` ``` Shut down ./remove-docker-swarm-service.sh --image-name "dataartisans/flink" my-flink-service ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/jgrier/flink release-1.2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3205.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3205 commit d380e98b397c770ee4b7eb9e4abb2387e7672d4c Author: Jamie Grier <jamie.grier@gmail.com> Date: 2017-01-21T00:12:17Z Improvements for Docker on Flink experience. Modifying Dockerfile to build from local flink-dist as well as release URLs. Logging to stdout. Adding scripts to deploy seamlessly on Docker Swarm. Updating Docker Compose scripts to work correctly. Parameterizing things so these Docker scripts are more generally useful. commit 962290ab66d993cac4c4e233a017feb8a6eb1cc5 Author: Jamie Grier <jamie.grier@gmail.com> Date: 2017-01-21T00:14:27Z Adding flag to flink-daemon.sh and related scripts so that we can actually log to the console – which is better for Docker environments. commit 66c2133ede0ed9914baba2fe7b93f1b9d585ede2 Author: Jamie Grier <jamie.grier@gmail.com> Date: 2017-01-25T05:45:23Z Adding Apache license to new files.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ex00 commented on the issue:

          https://github.com/apache/flink/pull/3205

          Hi @jgrier. Thaks for your PR!
          Could you update [README.md](https://github.com/apache/flink/blob/master/flink-contrib/docker-flink/README.md) in accordance with your changes?

          Show
          githubbot ASF GitHub Bot added a comment - Github user ex00 commented on the issue: https://github.com/apache/flink/pull/3205 Hi @jgrier. Thaks for your PR! Could you update [README.md] ( https://github.com/apache/flink/blob/master/flink-contrib/docker-flink/README.md ) in accordance with your changes?
          Hide
          greghogan Greg Hogan added a comment -

          Jamie Grier, will this subsume FLINK-4326?

          Show
          greghogan Greg Hogan added a comment - Jamie Grier , will this subsume FLINK-4326 ?
          Hide
          jgrier Jamie Grier added a comment -

          Greg Hogan I actually wasn't aware of FLINK-4326. However there does seem to be some overlap. There is also FLINK-5634 and an associated PR which specifically addresses enabling logging to stdout rather than to a file. However, neither of these two issues addresses running Flink processes in the foreground and avoiding forking as in FLINK-4326. I think this remains a separate issue that may need to be addressed.

          My main motivation in both of these JIRA issues and associated PRs was in providing a better Flink on Docker experience and providing some example scripts of how to run Flink properly in container-based environments. I would also like to get some "official" Flink Docker images published once we're satisfied with them.

          Show
          jgrier Jamie Grier added a comment - Greg Hogan I actually wasn't aware of FLINK-4326 . However there does seem to be some overlap. There is also FLINK-5634 and an associated PR which specifically addresses enabling logging to stdout rather than to a file. However, neither of these two issues addresses running Flink processes in the foreground and avoiding forking as in FLINK-4326 . I think this remains a separate issue that may need to be addressed. My main motivation in both of these JIRA issues and associated PRs was in providing a better Flink on Docker experience and providing some example scripts of how to run Flink properly in container-based environments. I would also like to get some "official" Flink Docker images published once we're satisfied with them.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jgrier commented on the issue:

          https://github.com/apache/flink/pull/3205

          Thanks @ex00. I missed that. Actually there is a new Flink on Docker stub in the top-level docs and I was planning to document these scripts there, however we may want to keep this here and just link to it. Anyway, I will update the docs. Thanks.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jgrier commented on the issue: https://github.com/apache/flink/pull/3205 Thanks @ex00. I missed that. Actually there is a new Flink on Docker stub in the top-level docs and I was planning to document these scripts there, however we may want to keep this here and just link to it. Anyway, I will update the docs. Thanks.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ex00 commented on the issue:

          https://github.com/apache/flink/pull/3205

          @jgrier thanks for your reply!
          I tried to build flink image from your PR and i got next error:

          >tar: invalid magic
          tar: short read
          The command '/bin/sh -c set -x && mkdir -p $FLINK_INSTALL_PATH && apk --update add --virtual build-dependencies curl &&
          curl -s $(curl -s https://www.apache.org/dyn/closer.cgi&#63;preferred\=true)flink/flink-$

          {FLINK_VERSION}

          /flink-$

          {FLINK_VERSIO N}

          -bin-hadoop$

          {HADOOP_VERSION}

          -scala_$

          {SCALA_VERSION}

          .tgz | tar xvz -C $FLINK_INSTALL_PATH && ln -s $FLINK_INSTALL_PATH/
          flink-$FLINK_VERSION $FLINK_HOME && addgroup -S flink && adduser -D -S -H -G flink -h $FLINK_HOME flink && chown -R flin
          k:flink $FLINK_INSTALL_PATH/flink-$FLINK_VERSION && chown -h flink:flink $FLINK_HOME && sed -i -e "s/echo \$mypid >> \$p
          id/echo \$mypid >> \$pid \&\& wait/g" $FLINK_HOME/bin/flink-daemon.sh && apk del build-dependencies && rm -rf /var/cache
          /apk/*' returned a non-zero code: 1

          full error [here](https://gist.github.com/ex00/5246c1df60e51c3a3581eaa6bc683176)
          My env:
          ```
          $ docker version
          Client:
          Version: 1.11.1
          API version: 1.23
          Go version: go1.5.4
          Git commit: 5604cbe
          Built: Tue Apr 26 23:44:17 2016
          OS/Arch: windows/amd64

          Server:
          Version: 1.11.2
          API version: 1.23
          Go version: go1.5.4
          Git commit: b9f10c9
          Built: Wed Jun 1 21:20:08 2016
          OS/Arch: linux/amd64
          ```
          Did you have look like error?

          Show
          githubbot ASF GitHub Bot added a comment - Github user ex00 commented on the issue: https://github.com/apache/flink/pull/3205 @jgrier thanks for your reply! I tried to build flink image from your PR and i got next error: >tar: invalid magic tar: short read The command '/bin/sh -c set -x && mkdir -p $FLINK_INSTALL_PATH && apk --update add --virtual build-dependencies curl && curl -s $(curl -s https://www.apache.org/dyn/closer.cgi&#63;preferred\=true)flink/flink-$ {FLINK_VERSION} /flink-$ {FLINK_VERSIO N} -bin-hadoop$ {HADOOP_VERSION} -scala_$ {SCALA_VERSION} .tgz | tar xvz -C $FLINK_INSTALL_PATH && ln -s $FLINK_INSTALL_PATH/ flink-$FLINK_VERSION $FLINK_HOME && addgroup -S flink && adduser -D -S -H -G flink -h $FLINK_HOME flink && chown -R flin k:flink $FLINK_INSTALL_PATH/flink-$FLINK_VERSION && chown -h flink:flink $FLINK_HOME && sed -i -e "s/echo \$mypid >> \$p id/echo \$mypid >> \$pid \&\& wait/g" $FLINK_HOME/bin/flink-daemon.sh && apk del build-dependencies && rm -rf /var/cache /apk/*' returned a non-zero code: 1 full error [here] ( https://gist.github.com/ex00/5246c1df60e51c3a3581eaa6bc683176 ) My env: ``` $ docker version Client: Version: 1.11.1 API version: 1.23 Go version: go1.5.4 Git commit: 5604cbe Built: Tue Apr 26 23:44:17 2016 OS/Arch: windows/amd64 Server: Version: 1.11.2 API version: 1.23 Go version: go1.5.4 Git commit: b9f10c9 Built: Wed Jun 1 21:20:08 2016 OS/Arch: linux/amd64 ``` Did you have look like error?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jgrier commented on the issue:

          https://github.com/apache/flink/pull/3205

          @ex00 The `--from-local-dist` flag assumes you've already built Flink locally. If you run `mvn clean package -DskipTests` does this error persist?

          Show
          githubbot ASF GitHub Bot added a comment - Github user jgrier commented on the issue: https://github.com/apache/flink/pull/3205 @ex00 The `--from-local-dist` flag assumes you've already built Flink locally. If you run `mvn clean package -DskipTests` does this error persist?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ex00 commented on the issue:

          https://github.com/apache/flink/pull/3205

          Really I didn't build Flink. Thanks @jgrier for your comment. My case has been showing that documentation is needed

          I think next information will be useful for users who want run flink on docker machine of windows.
          Then I built image, i tried run docker-compose and I got next message
          ```
          $ docker-compose up -d
          Creating dockerflink_jobmanager_1
          ERROR: for jobmanager Container command 'docker-entrypoint.sh' not found or does not exist.
          ```
          I tried execute docker-entrypoint.sh manually in container
          ```
          bash-4.3$ /opt/flink/bin/docker-entrypoint.sh
          bash: /opt/flink/bin/docker-entrypoint.sh: /bin/sh^M: bad interpreter: No such file or directory
          ```
          I changed line end format for docker-entrypoint.sh on Unix and after that docker-compose has started successfully.

          Show
          githubbot ASF GitHub Bot added a comment - Github user ex00 commented on the issue: https://github.com/apache/flink/pull/3205 Really I didn't build Flink. Thanks @jgrier for your comment. My case has been showing that documentation is needed I think next information will be useful for users who want run flink on docker machine of windows. Then I built image, i tried run docker-compose and I got next message ``` $ docker-compose up -d Creating dockerflink_jobmanager_1 ERROR: for jobmanager Container command 'docker-entrypoint.sh' not found or does not exist. ``` I tried execute docker-entrypoint.sh manually in container ``` bash-4.3$ /opt/flink/bin/docker-entrypoint.sh bash: /opt/flink/bin/docker-entrypoint.sh: /bin/sh^M: bad interpreter: No such file or directory ``` I changed line end format for docker-entrypoint.sh on Unix and after that docker-compose has started successfully.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tonycox commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3205#discussion_r100148697

          — Diff: flink-contrib/docker-flink/Dockerfile —
          @@ -36,22 +31,25 @@ ENV PATH $PATH:$FLINK_HOME/bin
          EXPOSE 8081
          EXPOSE 6123

          +# flink-dist can point to a directory, a tarball on the local system, or a url to a tarball
          +ARG flink_dist=NOT_SET
          +

          1. Install build dependencies and flink
            +ADD $flink_dist $FLINK_INSTALL_PATH
            RUN set -x && \
            mkdir -p $FLINK_INSTALL_PATH && \
              • End diff –

          place of error. when you add `$flink_dist` path of `$FLINK_INSTALL_PATH` will create. I propose to delete making dir here

          Show
          githubbot ASF GitHub Bot added a comment - Github user tonycox commented on a diff in the pull request: https://github.com/apache/flink/pull/3205#discussion_r100148697 — Diff: flink-contrib/docker-flink/Dockerfile — @@ -36,22 +31,25 @@ ENV PATH $PATH:$FLINK_HOME/bin EXPOSE 8081 EXPOSE 6123 +# flink-dist can point to a directory, a tarball on the local system, or a url to a tarball +ARG flink_dist=NOT_SET + Install build dependencies and flink +ADD $flink_dist $FLINK_INSTALL_PATH RUN set -x && \ mkdir -p $FLINK_INSTALL_PATH && \ End diff – place of error. when you add `$flink_dist` path of `$FLINK_INSTALL_PATH` will create. I propose to delete making dir here
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tonycox commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3205#discussion_r100156082

          — Diff: flink-contrib/docker-flink/Dockerfile —
          @@ -36,22 +31,25 @@ ENV PATH $PATH:$FLINK_HOME/bin
          EXPOSE 8081
          EXPOSE 6123

          +# flink-dist can point to a directory, a tarball on the local system, or a url to a tarball
          +ARG flink_dist=NOT_SET
          +

          1. Install build dependencies and flink
            +ADD $flink_dist $FLINK_INSTALL_PATH
            RUN set -x && \
            mkdir -p $FLINK_INSTALL_PATH && \
              • End diff –

          and what if `$flink_dist` is url ?
          Should run `tar xzf` I think

          Show
          githubbot ASF GitHub Bot added a comment - Github user tonycox commented on a diff in the pull request: https://github.com/apache/flink/pull/3205#discussion_r100156082 — Diff: flink-contrib/docker-flink/Dockerfile — @@ -36,22 +31,25 @@ ENV PATH $PATH:$FLINK_HOME/bin EXPOSE 8081 EXPOSE 6123 +# flink-dist can point to a directory, a tarball on the local system, or a url to a tarball +ARG flink_dist=NOT_SET + Install build dependencies and flink +ADD $flink_dist $FLINK_INSTALL_PATH RUN set -x && \ mkdir -p $FLINK_INSTALL_PATH && \ End diff – and what if `$flink_dist` is url ? Should run `tar xzf` I think
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tonycox commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3205#discussion_r100159827

          — Diff: flink-contrib/docker-flink/docker-compose.yml —
          @@ -16,21 +16,22 @@

          1. limitations under the License.
            ################################################################################

          -version: "2"
          +# Set the FLINK_DOCKER_IMAGE_NAME environment variable to override the image name to use
          +
          +version: "2.1"
          services:
          jobmanager:

          • image: flink
          • container_name: "jobmanager"
            + image: $ {FLINK_DOCKER_IMAGE_NAME:-flink}

            expose:

          • "6123"
              • End diff –

          do we need this if it set in Dockerfile ?

          Show
          githubbot ASF GitHub Bot added a comment - Github user tonycox commented on a diff in the pull request: https://github.com/apache/flink/pull/3205#discussion_r100159827 — Diff: flink-contrib/docker-flink/docker-compose.yml — @@ -16,21 +16,22 @@ limitations under the License. ################################################################################ -version: "2" +# Set the FLINK_DOCKER_IMAGE_NAME environment variable to override the image name to use + +version: "2.1" services: jobmanager: image: flink container_name: "jobmanager" + image: $ {FLINK_DOCKER_IMAGE_NAME:-flink} expose: "6123" End diff – do we need this if it set in Dockerfile ?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ex00 commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3205#discussion_r100252102

          — Diff: flink-contrib/docker-flink/docker-compose.yml —
          @@ -16,21 +16,22 @@

          1. limitations under the License.
            ################################################################################

          -version: "2"
          +# Set the FLINK_DOCKER_IMAGE_NAME environment variable to override the image name to use
          +
          +version: "2.1"
          services:
          jobmanager:

          • image: flink
          • container_name: "jobmanager"
            + image: $ {FLINK_DOCKER_IMAGE_NAME:-flink}

            expose:

          • "6123"
              • End diff –

          the variable in yml doesn't depends variable from Dockerfile.
          see https://docs.docker.com/compose/compose-file/#variable-substitution

          Show
          githubbot ASF GitHub Bot added a comment - Github user ex00 commented on a diff in the pull request: https://github.com/apache/flink/pull/3205#discussion_r100252102 — Diff: flink-contrib/docker-flink/docker-compose.yml — @@ -16,21 +16,22 @@ limitations under the License. ################################################################################ -version: "2" +# Set the FLINK_DOCKER_IMAGE_NAME environment variable to override the image name to use + +version: "2.1" services: jobmanager: image: flink container_name: "jobmanager" + image: $ {FLINK_DOCKER_IMAGE_NAME:-flink} expose: "6123" End diff – the variable in yml doesn't depends variable from Dockerfile. see https://docs.docker.com/compose/compose-file/#variable-substitution
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user patricklucas opened a pull request:

          https://github.com/apache/flink/pull/3494

          FLINK-5635 [docker] Improve Docker tooling

          This PR supplants and extends @jgrier's #3205, updating with master, fixing merge conflicts, making use of the new `start-foreground` param, and applying some review feedback from #3205.

          When merged, #3205 can be closed.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/patricklucas/flink FLINK-5635_improve_docker_tooling

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3494.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3494


          commit 1ca6229e71f3dad7c87d9d0a7c759998af8258f5
          Author: Jamie Grier <jamie.grier@gmail.com>
          Date: 2017-01-21T00:12:17Z

          Improvements for Docker on Flink experience.

          Modifying Dockerfile to build from local flink-dist as well as release URLs.
          Logging to stdout.
          Adding scripts to deploy seamlessly on Docker Swarm.
          Updating Docker Compose scripts to work correctly.
          Parameterizing things so these Docker scripts are more generally useful.

          commit 8bad184d3a77259d84ebe4b34c8c28d61df5c8e3
          Author: Jamie Grier <jamie.grier@gmail.com>
          Date: 2017-01-25T05:45:23Z

          Adding Apache license to new files.

          commit afb8a71596d2e833cd905bc6669418b74da75b61
          Author: Patrick Lucas <me@patricklucas.com>
          Date: 2017-03-07T15:26:12Z

          Use start-foreground in Docker entrypoint

          commit 13f7826b9403ddb376716019a9142d869e559781
          Author: Patrick Lucas <me@patricklucas.com>
          Date: 2017-03-07T16:22:21Z

          docker-entrypoint.sh should error on invalid args

          commit 305796df1c9f5bc81ed3e320c0aba7afed5a1d61
          Author: Patrick Lucas <me@patricklucas.com>
          Date: 2017-03-07T16:39:35Z

          Improve docker build.sh cleanup

          commit 545226dfbbc9a2248f34421d3ef1de572b9ad5f7
          Author: Patrick Lucas <me@patricklucas.com>
          Date: 2017-03-08T10:51:12Z

          Dockerfile improvements per review


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user patricklucas opened a pull request: https://github.com/apache/flink/pull/3494 FLINK-5635 [docker] Improve Docker tooling This PR supplants and extends @jgrier's #3205, updating with master, fixing merge conflicts, making use of the new `start-foreground` param, and applying some review feedback from #3205. When merged, #3205 can be closed. You can merge this pull request into a Git repository by running: $ git pull https://github.com/patricklucas/flink FLINK-5635 _improve_docker_tooling Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3494.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3494 commit 1ca6229e71f3dad7c87d9d0a7c759998af8258f5 Author: Jamie Grier <jamie.grier@gmail.com> Date: 2017-01-21T00:12:17Z Improvements for Docker on Flink experience. Modifying Dockerfile to build from local flink-dist as well as release URLs. Logging to stdout. Adding scripts to deploy seamlessly on Docker Swarm. Updating Docker Compose scripts to work correctly. Parameterizing things so these Docker scripts are more generally useful. commit 8bad184d3a77259d84ebe4b34c8c28d61df5c8e3 Author: Jamie Grier <jamie.grier@gmail.com> Date: 2017-01-25T05:45:23Z Adding Apache license to new files. commit afb8a71596d2e833cd905bc6669418b74da75b61 Author: Patrick Lucas <me@patricklucas.com> Date: 2017-03-07T15:26:12Z Use start-foreground in Docker entrypoint commit 13f7826b9403ddb376716019a9142d869e559781 Author: Patrick Lucas <me@patricklucas.com> Date: 2017-03-07T16:22:21Z docker-entrypoint.sh should error on invalid args commit 305796df1c9f5bc81ed3e320c0aba7afed5a1d61 Author: Patrick Lucas <me@patricklucas.com> Date: 2017-03-07T16:39:35Z Improve docker build.sh cleanup commit 545226dfbbc9a2248f34421d3ef1de572b9ad5f7 Author: Patrick Lucas <me@patricklucas.com> Date: 2017-03-08T10:51:12Z Dockerfile improvements per review
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user patricklucas commented on the issue:

          https://github.com/apache/flink/pull/3494

          cc @ex00 @tonycox @iemejia

          Show
          githubbot ASF GitHub Bot added a comment - Github user patricklucas commented on the issue: https://github.com/apache/flink/pull/3494 cc @ex00 @tonycox @iemejia
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user iemejia commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3494#discussion_r104907373

          — Diff: flink-contrib/docker-flink/Dockerfile —
          @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin
          EXPOSE 8081
          EXPOSE 6123

          +# flink-dist can point to a directory or a tarball on the local system
          +ARG flink_dist=NOT_SET
          +

          1. Install build dependencies and flink
            +ADD $flink_dist $FLINK_INSTALL_PATH
            RUN set -x && \

          We will break compatibility with docker hub if we move this out. Docker hub does not invoke a shell script but an specific Dockerfile hosted in a github repo, so this should not be done, or at least we should validate that we can build the image from docker hub.

          Show
          githubbot ASF GitHub Bot added a comment - Github user iemejia commented on a diff in the pull request: https://github.com/apache/flink/pull/3494#discussion_r104907373 — Diff: flink-contrib/docker-flink/Dockerfile — @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin EXPOSE 8081 EXPOSE 6123 +# flink-dist can point to a directory or a tarball on the local system +ARG flink_dist=NOT_SET + Install build dependencies and flink +ADD $flink_dist $FLINK_INSTALL_PATH RUN set -x && \ mkdir -p $FLINK_INSTALL_PATH && \ apk --update add --virtual build-dependencies curl && \ curl -s $(curl -s https://www.apache.org/dyn/closer.cgi&#63;preferred\=true)flink/flink-$ {FLINK_VERSION}/flink-${FLINK_VERSION} -bin-hadoop$ {HADOOP_VERSION} -scala_$ {SCALA_VERSION} .tgz | \ End diff – We will break compatibility with docker hub if we move this out. Docker hub does not invoke a shell script but an specific Dockerfile hosted in a github repo, so this should not be done, or at least we should validate that we can build the image from docker hub.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user patricklucas commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3494#discussion_r104912987

          — Diff: flink-contrib/docker-flink/Dockerfile —
          @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin
          EXPOSE 8081
          EXPOSE 6123

          +# flink-dist can point to a directory or a tarball on the local system
          +ARG flink_dist=NOT_SET
          +

          1. Install build dependencies and flink
            +ADD $flink_dist $FLINK_INSTALL_PATH
            RUN set -x && \

          That makes sense, I'll take a look at other official repos and come up with something that preserves the functionality we're trying to have here.

          Show
          githubbot ASF GitHub Bot added a comment - Github user patricklucas commented on a diff in the pull request: https://github.com/apache/flink/pull/3494#discussion_r104912987 — Diff: flink-contrib/docker-flink/Dockerfile — @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin EXPOSE 8081 EXPOSE 6123 +# flink-dist can point to a directory or a tarball on the local system +ARG flink_dist=NOT_SET + Install build dependencies and flink +ADD $flink_dist $FLINK_INSTALL_PATH RUN set -x && \ mkdir -p $FLINK_INSTALL_PATH && \ apk --update add --virtual build-dependencies curl && \ curl -s $(curl -s https://www.apache.org/dyn/closer.cgi&#63;preferred\=true)flink/flink-$ {FLINK_VERSION}/flink-${FLINK_VERSION} -bin-hadoop$ {HADOOP_VERSION} -scala_$ {SCALA_VERSION} .tgz | \ End diff – That makes sense, I'll take a look at other official repos and come up with something that preserves the functionality we're trying to have here.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user iemejia commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3494#discussion_r104919360

          — Diff: flink-contrib/docker-flink/Dockerfile —
          @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin
          EXPOSE 8081
          EXPOSE 6123

          +# flink-dist can point to a directory or a tarball on the local system
          +ARG flink_dist=NOT_SET
          +

          1. Install build dependencies and flink
            +ADD $flink_dist $FLINK_INSTALL_PATH
            RUN set -x && \

          The easiest way to do this is to create a docker hub account for this and try to build the image from your own fork of flink, that's what I did before to test it:
          https://hub.docker.com/r/iemejia/flink/

          Show
          githubbot ASF GitHub Bot added a comment - Github user iemejia commented on a diff in the pull request: https://github.com/apache/flink/pull/3494#discussion_r104919360 — Diff: flink-contrib/docker-flink/Dockerfile — @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin EXPOSE 8081 EXPOSE 6123 +# flink-dist can point to a directory or a tarball on the local system +ARG flink_dist=NOT_SET + Install build dependencies and flink +ADD $flink_dist $FLINK_INSTALL_PATH RUN set -x && \ mkdir -p $FLINK_INSTALL_PATH && \ apk --update add --virtual build-dependencies curl && \ curl -s $(curl -s https://www.apache.org/dyn/closer.cgi&#63;preferred\=true)flink/flink-$ {FLINK_VERSION}/flink-${FLINK_VERSION} -bin-hadoop$ {HADOOP_VERSION} -scala_$ {SCALA_VERSION} .tgz | \ End diff – The easiest way to do this is to create a docker hub account for this and try to build the image from your own fork of flink, that's what I did before to test it: https://hub.docker.com/r/iemejia/flink/
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user patricklucas commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3494#discussion_r104979246

          — Diff: flink-contrib/docker-flink/Dockerfile —
          @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin
          EXPOSE 8081
          EXPOSE 6123

          +# flink-dist can point to a directory or a tarball on the local system
          +ARG flink_dist=NOT_SET
          +

          1. Install build dependencies and flink
            +ADD $flink_dist $FLINK_INSTALL_PATH
            RUN set -x && \

          I spoke with @jgrier and we decided to take the route of creating a separate repo for the official images with all the Dockerfile variants we need, like most other projects have.

          In the mean time, I think we can go ahead and merge this change (and yours in #3493) as a general improvement for Flink developers and users who just want a nice way to create Docker images from a release or their own Flink repo.

          Show
          githubbot ASF GitHub Bot added a comment - Github user patricklucas commented on a diff in the pull request: https://github.com/apache/flink/pull/3494#discussion_r104979246 — Diff: flink-contrib/docker-flink/Dockerfile — @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin EXPOSE 8081 EXPOSE 6123 +# flink-dist can point to a directory or a tarball on the local system +ARG flink_dist=NOT_SET + Install build dependencies and flink +ADD $flink_dist $FLINK_INSTALL_PATH RUN set -x && \ mkdir -p $FLINK_INSTALL_PATH && \ apk --update add --virtual build-dependencies curl && \ curl -s $(curl -s https://www.apache.org/dyn/closer.cgi&#63;preferred\=true)flink/flink-$ {FLINK_VERSION}/flink-${FLINK_VERSION} -bin-hadoop$ {HADOOP_VERSION} -scala_$ {SCALA_VERSION} .tgz | \ End diff – I spoke with @jgrier and we decided to take the route of creating a separate repo for the official images with all the Dockerfile variants we need, like most other projects have. In the mean time, I think we can go ahead and merge this change (and yours in #3493) as a general improvement for Flink developers and users who just want a nice way to create Docker images from a release or their own Flink repo.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user iemejia commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3494#discussion_r104980420

          — Diff: flink-contrib/docker-flink/Dockerfile —
          @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin
          EXPOSE 8081
          EXPOSE 6123

          +# flink-dist can point to a directory or a tarball on the local system
          +ARG flink_dist=NOT_SET
          +

          1. Install build dependencies and flink
            +ADD $flink_dist $FLINK_INSTALL_PATH
            RUN set -x && \

          Pefect, makes sense, but we should probably keep updating the existing image for some time, because this is the way most people are aware that it exists, or add a README that it was moved and maintained in the new repo.
          Are you going to be the official maintainer (it is ok with me if you want to take the lead there, I just want to learn the whole process of creating an official image), but I can also help if you need with maintenance.

          Show
          githubbot ASF GitHub Bot added a comment - Github user iemejia commented on a diff in the pull request: https://github.com/apache/flink/pull/3494#discussion_r104980420 — Diff: flink-contrib/docker-flink/Dockerfile — @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin EXPOSE 8081 EXPOSE 6123 +# flink-dist can point to a directory or a tarball on the local system +ARG flink_dist=NOT_SET + Install build dependencies and flink +ADD $flink_dist $FLINK_INSTALL_PATH RUN set -x && \ mkdir -p $FLINK_INSTALL_PATH && \ apk --update add --virtual build-dependencies curl && \ curl -s $(curl -s https://www.apache.org/dyn/closer.cgi&#63;preferred\=true)flink/flink-$ {FLINK_VERSION}/flink-${FLINK_VERSION} -bin-hadoop$ {HADOOP_VERSION} -scala_$ {SCALA_VERSION} .tgz | \ End diff – Pefect, makes sense, but we should probably keep updating the existing image for some time, because this is the way most people are aware that it exists, or add a README that it was moved and maintained in the new repo. Are you going to be the official maintainer (it is ok with me if you want to take the lead there, I just want to learn the whole process of creating an official image), but I can also help if you need with maintenance.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user patricklucas commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3494#discussion_r104982158

          — Diff: flink-contrib/docker-flink/Dockerfile —
          @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin
          EXPOSE 8081
          EXPOSE 6123

          +# flink-dist can point to a directory or a tarball on the local system
          +ARG flink_dist=NOT_SET
          +

          1. Install build dependencies and flink
            +ADD $flink_dist $FLINK_INSTALL_PATH
            RUN set -x && \

          Which existing image are you referring to that we need to keep updated? I'm not aware of a Flink project- or data Artisans-maintained image on Docker Hub.

          I'm almost done with the basic functionality for the first Alpine image (and a script to template its Dockerfile) but there will be plenty of room for improvements. Once the new repo is live I'll ping you and you can take a look.

          I don't know if I need to be the "official maintainer", but the repo will live under the [data Artisans GitHub org](https://github.com/dataArtisans), so if not me someone at dA will likely handle contributions to that repo.

          Show
          githubbot ASF GitHub Bot added a comment - Github user patricklucas commented on a diff in the pull request: https://github.com/apache/flink/pull/3494#discussion_r104982158 — Diff: flink-contrib/docker-flink/Dockerfile — @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin EXPOSE 8081 EXPOSE 6123 +# flink-dist can point to a directory or a tarball on the local system +ARG flink_dist=NOT_SET + Install build dependencies and flink +ADD $flink_dist $FLINK_INSTALL_PATH RUN set -x && \ mkdir -p $FLINK_INSTALL_PATH && \ apk --update add --virtual build-dependencies curl && \ curl -s $(curl -s https://www.apache.org/dyn/closer.cgi&#63;preferred\=true)flink/flink-$ {FLINK_VERSION}/flink-${FLINK_VERSION} -bin-hadoop$ {HADOOP_VERSION} -scala_$ {SCALA_VERSION} .tgz | \ End diff – Which existing image are you referring to that we need to keep updated? I'm not aware of a Flink project- or data Artisans-maintained image on Docker Hub. I'm almost done with the basic functionality for the first Alpine image (and a script to template its Dockerfile) but there will be plenty of room for improvements. Once the new repo is live I'll ping you and you can take a look. I don't know if I need to be the "official maintainer", but the repo will live under the [data Artisans GitHub org] ( https://github.com/dataArtisans ), so if not me someone at dA will likely handle contributions to that repo.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user iemejia commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3494#discussion_r104984654

          — Diff: flink-contrib/docker-flink/Dockerfile —
          @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin
          EXPOSE 8081
          EXPOSE 6123

          +# flink-dist can point to a directory or a tarball on the local system
          +ARG flink_dist=NOT_SET
          +

          1. Install build dependencies and flink
            +ADD $flink_dist $FLINK_INSTALL_PATH
            RUN set -x && \

          I am not talking about a published image on docker hub, but the image that exists in the flink repo, the whole reason I jumped in and worked in this image was because I was aware that the community was around the image, and it will be improved/updated, people can't be aware of the image existance/maintenance if we don't show it somewhere.

          Ok for the image on dataArtisans it makes sense but I have to be honest I would prefer it to be still at Apache, at least I hope it preserves the license (I hadn't seen you work there, nice).

          And yes, when you propose an image, you have to show that you plan to maintain it, or that you are part of the community.
          https://github.com/docker-library/official-images#maintainership
          This is important because they (docker) want images to get security updates / improvements.

          Show
          githubbot ASF GitHub Bot added a comment - Github user iemejia commented on a diff in the pull request: https://github.com/apache/flink/pull/3494#discussion_r104984654 — Diff: flink-contrib/docker-flink/Dockerfile — @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin EXPOSE 8081 EXPOSE 6123 +# flink-dist can point to a directory or a tarball on the local system +ARG flink_dist=NOT_SET + Install build dependencies and flink +ADD $flink_dist $FLINK_INSTALL_PATH RUN set -x && \ mkdir -p $FLINK_INSTALL_PATH && \ apk --update add --virtual build-dependencies curl && \ curl -s $(curl -s https://www.apache.org/dyn/closer.cgi&#63;preferred\=true)flink/flink-$ {FLINK_VERSION}/flink-${FLINK_VERSION} -bin-hadoop$ {HADOOP_VERSION} -scala_$ {SCALA_VERSION} .tgz | \ End diff – I am not talking about a published image on docker hub, but the image that exists in the flink repo, the whole reason I jumped in and worked in this image was because I was aware that the community was around the image, and it will be improved/updated, people can't be aware of the image existance/maintenance if we don't show it somewhere. Ok for the image on dataArtisans it makes sense but I have to be honest I would prefer it to be still at Apache, at least I hope it preserves the license (I hadn't seen you work there, nice). And yes, when you propose an image, you have to show that you plan to maintain it, or that you are part of the community. https://github.com/docker-library/official-images#maintainership This is important because they (docker) want images to get security updates / improvements.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user patricklucas commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3494#discussion_r104986831

          — Diff: flink-contrib/docker-flink/Dockerfile —
          @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin
          EXPOSE 8081
          EXPOSE 6123

          +# flink-dist can point to a directory or a tarball on the local system
          +ARG flink_dist=NOT_SET
          +

          1. Install build dependencies and flink
            +ADD $flink_dist $FLINK_INSTALL_PATH
            RUN set -x && \

          I see, just a terminology thing: image refers to the binary blob Docker Hub stores for download, in the Flink repo is a Dockerfile for generating that image.

          This is mainly to follow suit with the other Apache projects I have found with official images; they all tend to host their Dockerfiles totally separately. Needless to say, the repo will be wholly licensed under the Apache license, just not maintained under the ASF.

          And we're definitely committed to maintaining these official images going forward, updating them as a part of the Flink release process. If they need an individual maintainer point of contact, that can be me and I can hand it off later if I have to.

          Show
          githubbot ASF GitHub Bot added a comment - Github user patricklucas commented on a diff in the pull request: https://github.com/apache/flink/pull/3494#discussion_r104986831 — Diff: flink-contrib/docker-flink/Dockerfile — @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin EXPOSE 8081 EXPOSE 6123 +# flink-dist can point to a directory or a tarball on the local system +ARG flink_dist=NOT_SET + Install build dependencies and flink +ADD $flink_dist $FLINK_INSTALL_PATH RUN set -x && \ mkdir -p $FLINK_INSTALL_PATH && \ apk --update add --virtual build-dependencies curl && \ curl -s $(curl -s https://www.apache.org/dyn/closer.cgi&#63;preferred\=true)flink/flink-$ {FLINK_VERSION}/flink-${FLINK_VERSION} -bin-hadoop$ {HADOOP_VERSION} -scala_$ {SCALA_VERSION} .tgz | \ End diff – I see, just a terminology thing: image refers to the binary blob Docker Hub stores for download, in the Flink repo is a Dockerfile for generating that image. This is mainly to follow suit with the other Apache projects I have found with official images; they all tend to host their Dockerfiles totally separately. Needless to say, the repo will be wholly licensed under the Apache license, just not maintained under the ASF. And we're definitely committed to maintaining these official images going forward, updating them as a part of the Flink release process. If they need an individual maintainer point of contact, that can be me and I can hand it off later if I have to.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jgrier commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3494#discussion_r105065007

          — Diff: flink-contrib/docker-flink/Dockerfile —
          @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin
          EXPOSE 8081
          EXPOSE 6123

          +# flink-dist can point to a directory or a tarball on the local system
          +ARG flink_dist=NOT_SET
          +

          1. Install build dependencies and flink
            +ADD $flink_dist $FLINK_INSTALL_PATH
            RUN set -x && \

          Hi guys, thanks for picking up this PR and running with it. I think we should separate the "official" Docker images issue from this PR. This PR was about just simply improving the existing Docker tooling and making it easier for Flink developers to build their own images, maybe with their own tweaks, or specifying specific Flink releases to build the image from.

          On the "official" images topic I'm totally fine with either approach. dA could do this and maintain those official images or we could do it as part of Flink and the community can maintain them. The main thing for me is just seeing that it's super easy for people to get Flink running with Docker and there is some accepted "official" image for people to grab rather than wading through a bunch of Flink images generated by various people. That's bad for the user experience.

          Regardless, let's figure out separately how best to get an official image on DockerHub and get this change finished up because I think the changes here are useful as is.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jgrier commented on a diff in the pull request: https://github.com/apache/flink/pull/3494#discussion_r105065007 — Diff: flink-contrib/docker-flink/Dockerfile — @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin EXPOSE 8081 EXPOSE 6123 +# flink-dist can point to a directory or a tarball on the local system +ARG flink_dist=NOT_SET + Install build dependencies and flink +ADD $flink_dist $FLINK_INSTALL_PATH RUN set -x && \ mkdir -p $FLINK_INSTALL_PATH && \ apk --update add --virtual build-dependencies curl && \ curl -s $(curl -s https://www.apache.org/dyn/closer.cgi&#63;preferred\=true)flink/flink-$ {FLINK_VERSION}/flink-${FLINK_VERSION} -bin-hadoop$ {HADOOP_VERSION} -scala_$ {SCALA_VERSION} .tgz | \ End diff – Hi guys, thanks for picking up this PR and running with it. I think we should separate the "official" Docker images issue from this PR. This PR was about just simply improving the existing Docker tooling and making it easier for Flink developers to build their own images, maybe with their own tweaks, or specifying specific Flink releases to build the image from. On the "official" images topic I'm totally fine with either approach. dA could do this and maintain those official images or we could do it as part of Flink and the community can maintain them. The main thing for me is just seeing that it's super easy for people to get Flink running with Docker and there is some accepted "official" image for people to grab rather than wading through a bunch of Flink images generated by various people. That's bad for the user experience. Regardless, let's figure out separately how best to get an official image on DockerHub and get this change finished up because I think the changes here are useful as is.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user patricklucas commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3494#discussion_r105119473

          — Diff: flink-contrib/docker-flink/Dockerfile —
          @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin
          EXPOSE 8081
          EXPOSE 6123

          +# flink-dist can point to a directory or a tarball on the local system
          +ARG flink_dist=NOT_SET
          +

          1. Install build dependencies and flink
            +ADD $flink_dist $FLINK_INSTALL_PATH
            RUN set -x && \

          I agree. I'll comment on FLINK-3026 with a summary of this discussion.

          Show
          githubbot ASF GitHub Bot added a comment - Github user patricklucas commented on a diff in the pull request: https://github.com/apache/flink/pull/3494#discussion_r105119473 — Diff: flink-contrib/docker-flink/Dockerfile — @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin EXPOSE 8081 EXPOSE 6123 +# flink-dist can point to a directory or a tarball on the local system +ARG flink_dist=NOT_SET + Install build dependencies and flink +ADD $flink_dist $FLINK_INSTALL_PATH RUN set -x && \ mkdir -p $FLINK_INSTALL_PATH && \ apk --update add --virtual build-dependencies curl && \ curl -s $(curl -s https://www.apache.org/dyn/closer.cgi&#63;preferred\=true)flink/flink-$ {FLINK_VERSION}/flink-${FLINK_VERSION} -bin-hadoop$ {HADOOP_VERSION} -scala_$ {SCALA_VERSION} .tgz | \ End diff – I agree. I'll comment on FLINK-3026 with a summary of this discussion.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user iemejia commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3494#discussion_r105171729

          — Diff: flink-contrib/docker-flink/Dockerfile —
          @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin
          EXPOSE 8081
          EXPOSE 6123

          +# flink-dist can point to a directory or a tarball on the local system
          +ARG flink_dist=NOT_SET
          +

          1. Install build dependencies and flink
            +ADD $flink_dist $FLINK_INSTALL_PATH
            RUN set -x && \

          Hi, yes @jgrier the good thing is that we are all aligned on this, in wanting flink developers and users to have the best experience possible. And this is a discussion to follow in the other JIRA.

          Show
          githubbot ASF GitHub Bot added a comment - Github user iemejia commented on a diff in the pull request: https://github.com/apache/flink/pull/3494#discussion_r105171729 — Diff: flink-contrib/docker-flink/Dockerfile — @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin EXPOSE 8081 EXPOSE 6123 +# flink-dist can point to a directory or a tarball on the local system +ARG flink_dist=NOT_SET + Install build dependencies and flink +ADD $flink_dist $FLINK_INSTALL_PATH RUN set -x && \ mkdir -p $FLINK_INSTALL_PATH && \ apk --update add --virtual build-dependencies curl && \ curl -s $(curl -s https://www.apache.org/dyn/closer.cgi&#63;preferred\=true)flink/flink-$ {FLINK_VERSION}/flink-${FLINK_VERSION} -bin-hadoop$ {HADOOP_VERSION} -scala_$ {SCALA_VERSION} .tgz | \ End diff – Hi, yes @jgrier the good thing is that we are all aligned on this, in wanting flink developers and users to have the best experience possible. And this is a discussion to follow in the other JIRA.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user iemejia commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3494#discussion_r105171748

          — Diff: flink-contrib/docker-flink/Dockerfile —
          @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin
          EXPOSE 8081
          EXPOSE 6123

          +# flink-dist can point to a directory or a tarball on the local system
          +ARG flink_dist=NOT_SET
          +

          1. Install build dependencies and flink
            +ADD $flink_dist $FLINK_INSTALL_PATH
            RUN set -x && \

          @patricklucas I'd prefer to prioritize the other JIRA since this is more important (having the official image available), these cleanups we can do them afterwards. And maybe try to go ahead with #3500.

          Show
          githubbot ASF GitHub Bot added a comment - Github user iemejia commented on a diff in the pull request: https://github.com/apache/flink/pull/3494#discussion_r105171748 — Diff: flink-contrib/docker-flink/Dockerfile — @@ -36,22 +31,24 @@ ENV PATH $PATH:$FLINK_HOME/bin EXPOSE 8081 EXPOSE 6123 +# flink-dist can point to a directory or a tarball on the local system +ARG flink_dist=NOT_SET + Install build dependencies and flink +ADD $flink_dist $FLINK_INSTALL_PATH RUN set -x && \ mkdir -p $FLINK_INSTALL_PATH && \ apk --update add --virtual build-dependencies curl && \ curl -s $(curl -s https://www.apache.org/dyn/closer.cgi&#63;preferred\=true)flink/flink-$ {FLINK_VERSION}/flink-${FLINK_VERSION} -bin-hadoop$ {HADOOP_VERSION} -scala_$ {SCALA_VERSION} .tgz | \ End diff – @patricklucas I'd prefer to prioritize the other JIRA since this is more important (having the official image available), these cleanups we can do them afterwards. And maybe try to go ahead with #3500.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user uce commented on the issue:

          https://github.com/apache/flink/pull/3494

          Thanks for this It was interesting to follow your discussions @iemejia, @jgrier and @patricklucas. I've tried this out and it works as expected. I'm going to merge this later today if there are no objections. My understanding is that after merging this PR we can continue with #3500. If I misunderstood anything, please ping me here.

          Show
          githubbot ASF GitHub Bot added a comment - Github user uce commented on the issue: https://github.com/apache/flink/pull/3494 Thanks for this It was interesting to follow your discussions @iemejia, @jgrier and @patricklucas. I've tried this out and it works as expected. I'm going to merge this later today if there are no objections. My understanding is that after merging this PR we can continue with #3500. If I misunderstood anything, please ping me here.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3205

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3205
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3494

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3494
          Hide
          uce Ufuk Celebi added a comment -

          Fixed in 227478b, a3627f2 (master).

          Show
          uce Ufuk Celebi added a comment - Fixed in 227478b, a3627f2 (master).
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user iemejia commented on the issue:

          https://github.com/apache/flink/pull/3494

          Great, thanks @uce , I will take rebase an push the fix for FLINK-6003 (#3500) now.
          I will put you as a reviewer for this one.

          Show
          githubbot ASF GitHub Bot added a comment - Github user iemejia commented on the issue: https://github.com/apache/flink/pull/3494 Great, thanks @uce , I will take rebase an push the fix for FLINK-6003 (#3500) now. I will put you as a reviewer for this one.

            People

            • Assignee:
              plucas Patrick Lucas
              Reporter:
              jgrier Jamie Grier
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development