[BEAM-7474] Add SDK harness containers for Py 3.6, Py 3.7

• Type: Sub-task
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.16.0
• Component/s: sdk-py-harness
• Assignee: Hannah Jiang
• Reporter: Valentyn Tymofieiev
• Created: 01/Jun/19 01:26
• Updated: 24/Jul/20 19:54
• Resolved: 09/Sep/19 20:54

Description

Currently we can build a Py3-compatible container image with gradle by running:

./gradlew :sdks:python:container:py3:docker

This builds a docker container image like: valentyn-docker-apache.bintray.io/beam/python3

The code for this is defined in: https://github.com/apache/beam/blob/ae60a72b03f3a2b6b2a06667ec1868a7acc8e38f/sdks/python/container/py3/build.gradle#L48

To support portable runners that use a container (e.g. Flink) on multiple versions of Python 3, we should make it possible to build Python 3-compatible SDK harness containers bundled with any desired python version.

We could have several gradle projects:

:sdks:python:container:py35:docker
:sdks:python:container:py36:docker
:sdks:python:container:py37:docker

and several Dockerfiles to support this:

sdks/python/container/py35/Dockerfile
sdks/python/container/py36/Dockerfile
sdks/python/container/py37/Dockerfile

The only difference right now would be the base image used in FROM field in Dockerfile.

Alternatively, we could have one parameterized Dockerfile that starts with :

ARG BASE_IMAGE
FROM $BASE_IMAGE
...

I think the latter approach, may result in complications later if these containers will need to diverge down the road.

cc'ing a few folks who may have some feedback on this: angoenka mxm robertwb Juta frederik.

Issue Links

• is part of: BEAM-7907 Support customized container