[SPARK-35353] Cross-building docker images to ARM64 is failing (with Ubuntu host)

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 3.1.1
• Component/s: Kubernetes
• Assignee: Unassigned
• Reporter: Andy Grove
• Shepherd: Holden Karau
• Created: 08/May/21 16:02
• Updated: 20/Jul/21 16:47
• Resolved: 20/Jul/21 16:47

Description

I was trying to cross-build Spark 3.1.1 for ARM64 so that I could deploy to a Raspberry Pi Kubernetes cluster this weekend and the Docker build fails.

Here are the commands I used:

docker buildx create --use
./bin/docker-image-tool.sh -n -r andygrove -t 3.1.1 -X build 

The Docker build for ARM64 fails on the following command:

 apt install -y bash tini libc6 libpam-modules krb5-user libnss3 procps

The install fails with "Error while loading /usr/sbin/dpkg-split: No such file or directory".

Here is a fragment of the output showing the relevant error message.

#6 6.034 Get:35 https://deb.debian.org/debian buster/main arm64 libnss3 arm64 2:3.42.1-1+deb10u3 [1082 kB]
#6 6.102 Get:36 https://deb.debian.org/debian buster/main arm64 psmisc arm64 23.2-1 [122 kB]
#6 6.109 Get:37 https://deb.debian.org/debian buster/main arm64 tini arm64 0.18.0-1 [194 kB]
#6 6.767 debconf: delaying package configuration, since apt-utils is not installed
#6 6.883 Fetched 18.1 MB in 1s (13.4 MB/s)
#6 6.956 Error while loading /usr/sbin/dpkg-split: No such file or directory
#6 6.959 Error while loading /usr/sbin/dpkg-deb: No such file or directory
#6 6.961 dpkg: error processing archive /tmp/apt-dpkg-install-NdOR40/00-libncurses6_6.1+20181013-2+deb10u2_arm64.deb (--unpack):
 

My host environment details:

• Ubuntu 18.04.5 LTS
• Docker version 20.10.6, build 370c289