[IMPALA-7947] Run impalad minicluster processes inside docker containers

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 3.3.0
• Component/s: Infrastructure
• Labels: docker
• Target Version: Impala 3.3.0
• Epic Color: ghx-label-7
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 10/Dec/18 18:31
• Updated: 17/Aug/19 23:03
• Resolved: 17/Aug/19 23:03

Description

I'm looking on some docker-related tasks, e.g. IMPALA-7941 and it'd be cool to be able to start up a minicluster with each impala process inside its own container. This is distinct from test-with-docker which runs the whole minicluster inside a container and is much closer to how containerised services are deployed in production.

I'm imagining this would be a start-impala-cluster.py flag.

Sub-Tasks

• Create docker container for impalad/statestored/catalogd: Resolved
• Figure out how to handle localtime in docker minicluster containers: Resolved
• Extend start-impala-cluster.py to start and stop daemon docker containers: Resolved
• Get start-impala-cluster.py to start up a usable minicluster: Resolved
• Support loading data into a dockerised minicluster: Resolved
• Create coordinator and executor containers: Resolved
• Sort out what to do with bin/start-*d.sh functionality: Resolved
• Run end-to-end tests against docker in minicluster: Resolved
• Clean up config files in docker containers: Resolved
• Improve experience for configuring JVM heap size in docker container.: Resolved
• Pick better default flags in containers: Resolved
• Automate setup of docker bridge network for dockerised minicluster: Resolved
• make -j $IMPALA_BUILD_THREADS docker_images fails with OSError: [Errno 17] File exists: '/home/ubuntu/Impala/docker/build_context': Resolved
• Run container as non-root user: Resolved
• Re-enable large_string tests disabled for JVM OOM: Resolved
• Enable data cache by default for end-to-end containerised tests: Resolved
• Collect logs from docker containers in tests: Resolved
• Generate list of docker images generated by the build: Resolved
• Expose HS2 HTTP port in containers: Resolved
• Docker build should not depend on test config: Resolved
• container entrypoint should exec binary: Resolved
• --rpc_use_loopback may not work for runtime filter RPCs: Resolved
• Cmake dependencies for container build do not work correctly: Resolved
• Script to automate pushing images to docker repo, e.g. dockerhub: Resolved
• Re-enable catalog v2 in containers: Resolved