Details
-
Task
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
This is the uber jira for arm64 support in Ozone docker development environment.
Status Quo:
- Most images used in docker dev environment supports amd64 only.
- It runs VERY slowly on Apple Silicon (M1) Macs' Docker Desktop due to qemu x86-64 -> arm64 translation. For instance, SCM startup in the upgrade acceptance test should take ~23 seconds on a regular Intel MacBook Pro to boot up and exit safe mode; while it times out (after 240s~300s) starting up the SCM alone on an M1 Pro MacBook Pro due to the translation/emulation.
Goal:
- Add arm64 support for all Docker images used for Ozone development. The resulting images (that are pushed to Docker Hub) should be dual-architecture (e.g. ozone-docker-runner).
Foreseeable work to be done:
- Bump Centos base image to a version that supports arm64 (Done)
- dumbinit and tools included in the image build stage should be architecture aware. Currently the tools are x64 only.
- After that the ozone runner base image should be working, provided that ALL the Ozone dependencies at runtime that uses native library pulled from maven supports Linux arm64/aarch64 (afaik RocksDBJNI does, gRPC also does, newer protobuf 3 should also have it, but older protobuf 2.5.0 might not. However, it seems gRPC and protobuf 2/3 doesn't matter at the moment as they are only used at compile time but not runtime.)
- Have arm64 version of Hadoop 3 / KMS / KDC images as well, which are referenced in some testing environment like ozonesecure for KDC.
- Add new CI check when someone pulls in new Docker image that warns if the new image doesn't have the linux/arm64 architecture.
- The last point might not be strictly required as IIRC Docker itself supports running some containers natively but others in emulation at the same time.
- Hadoop 3 already has the arm64 Dockerfile ready. But no one has built and pushed an arm64 image to Docker Hub yet:
- https://hub.docker.com/r/apache/hadoop/tags
- https://hub.docker.com/r/apache/hadoop-runner/tags
