Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-40513

SPIP: Support Docker Official Image for Spark

    XMLWordPrintableJSON

Details

    Description

      This SPIP is proposed to add Docker Official Image(DOI) to ensure the Spark Docker images meet the quality standards for Docker images, to provide these Docker images for users who want to use Apache Spark via Docker image.

      There are also several Apache projects that release the Docker Official Images, such as: flink, storm, solr, zookeeper, httpd (with 50M+ to 1B+ download for each). From the huge download statistics, we can see the real demands of users, and from the support of other apache projects, we should also be able to do it.

      After support:

      • The Dockerfile will still be maintained by the Apache Spark community and reviewed by Docker.
      • The images will be maintained by the Docker community to ensure the quality standards for Docker images of the Docker community.

      It will also reduce the extra docker images maintenance effort (such as frequently rebuilding, image security update) of the Apache Spark community.

       

      SPIP DOC: https://docs.google.com/document/d/1nN-pKuvt-amUcrkTvYAQ-bJBgtsWb9nAkNoVNRM2S2o

      DISCUSS: https://lists.apache.org/thread/l1793y5224n8bqkp3s6ltgkykso4htb3

      Attachments

        1.
        		 Add apache/spark-docker repo Sub-task Resolved Matei Alexandru Zaharia
        2.
        		 Add .asf.yaml to apache/spark-docker Sub-task Resolved Yikun Jiang
        3.
        		 Add merge_spark_docker_pr.py to help merge commit Sub-task Resolved Yikun Jiang
        4.
        		 Add official image dockerfile for Spark v3.3.0 Sub-task Resolved Yikun Jiang
        5.
        		 Add PULL_REQUEST_TEMPLATE for spark-docker Sub-task Resolved Yikun Jiang
        6.
        		 Make Dockerfile build workflow work in apache repo Sub-task Resolved Yikun Jiang
        7.
        		 Enable Spark on K8s integration test for official dockerfiles Sub-task Resolved Yikun Jiang
        8.
        		 Add LICENSE and NOTICE for apache/spark-docker Sub-task Resolved Yikun Jiang
        9.
        		 Use `spark` username in official image Sub-task Resolved Yikun Jiang
        10.
        		 Add dockerfile template Sub-task Resolved Yikun Jiang
        11.
        		 Add README for spark-docker Sub-task Resolved Yikun Jiang
        12.
        		 Add template support for SPARK_GPG_KEY Sub-task Resolved Yikun Jiang
        13.
        		 Cleanup apt lists cache in Dockerfile Sub-task Resolved Yikun Jiang
        14.
        		 Rename Check Spark repo as Check Spark Docker repo in GA Sub-task Resolved Qian Sun
        15.
        		 Remove pip/setuptools dynamic upgrade Sub-task Resolved Yikun Jiang
        16.
        		 Add CONTRIBUTING.md to apache/spark-docker Sub-task Resolved Qian Sun
        17.
        		 Add smoke test in standalone cluster for spark-docker Sub-task Resolved Qian Sun
        18.
        		 Add official image dockerfile for Spark v3.3.1 Sub-task Resolved Yikun Jiang
        19.
        		 Add "Publish workflow" to help release apache/spark image Sub-task Resolved Unassigned
        20.
        		 Move image matrix into version's workflow Sub-task Resolved Yikun Jiang
        21.
        		 Add a test workflow to help test image in fork repo Sub-task Resolved Yikun Jiang
        22.
        		 Add a script to generate DOI mainifest Sub-task Resolved Yikun Jiang
        23.
        		 Add official image Dockerfile for Spark v3.3.2 Sub-task Resolved Yikun Jiang
        24.
        		 Add DOI manifest file for Spark Docker Official Image Sub-task In Progress Yikun Jiang
        25.
        		 Add Spark Docker Official Image doc Sub-task Open Unassigned
        26.
        		 Add doc for Docker Setup in standalone mode Sub-task Open Qian Sun
        27.
        		 Apply entrypoint template change to 3.3.0/3.3.1 Sub-task Resolved Yikun Jiang
        28.
        		 Add official image dockerfile for Spark v3.4.0 Sub-task Resolved Yikun Jiang
        29.
        		 Add apache/spark docker image overview Sub-task Open Unassigned
        30.
        		 Add Java17 dockerfiles for 3.5.0 Sub-task Resolved Yikun Jiang
        31.
        		 Refactor Dockerfile and workflow based on base image Sub-task Resolved Yikun Jiang
        32.
        		 Address DOI comments about /etc/passwd Sub-task Resolved Yikun Jiang
        33.
        		 Address comments about /etc/pam.d/su Sub-task Open Unassigned
        34.
        		 Recover sh support in Dockerfiles Sub-task Resolved Yikun Jiang
        35.
        		 Switch spark user only when run driver and executor Sub-task Resolved Yikun Jiang
        36.
        		 minimize duplication across layers for chmod Sub-task Resolved Yikun Jiang
        37.
        		 Use ; instead of && when enable set -ex Sub-task Resolved Yikun Jiang
        38.
        		 Fix SPARK_EXECUTOR_JAVA_OPTS assignment bug Sub-task Resolved Yikun Jiang
        39.
        		 Add awesome-spark-docker.md Sub-task Resolved Yikun Jiang
        40.
        		 Add Apache Spark 3.4.1 Dockerfiles Sub-task Resolved Yikun Jiang
        41.
        		 Remove useless lib64 path link in dockerfile Sub-task Resolved Unassigned
        42.
        		 Change apt to apt-get and remove useless cleanup Sub-task Resolved Yikun Jiang
        43.
        		 Add 'set -eo pipefail' to entrypoint and quote variables Sub-task Resolved Yikun Jiang
        44.
        		 Add --batch to gpg command Sub-task Resolved Unassigned
        45.
        		 Add official image Dockerfile for Spark 3.3.3 Sub-task Resolved Yuming Wang
        46.
        		 Add official image Dockerfile for Apache Spark 3.5.0 Sub-task Resolved Unassigned
        47.
        		 Add Java21 dockerfiles for 3.5.0 Sub-task Open Unassigned

        Activity

          People

            yikunkero Yikun Jiang
            yikunkero Yikun Jiang
            hyukjin.kwon hyukjin.kwon
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: