[SPARK-28347] Error loading glibc library when generatting data on HDFS in K8S

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.4.3
• Component/s: Kubernetes, Spark Core
• Assignee: Unassigned
• Reporter: Jiajia Li
• Created: 11/Jul/19 07:56
• Updated: 17/May/20 18:26
• Resolved: 12/Jul/19 05:59

Description

I want to run TPC-DS in K8S, when generating data on HDFS, with following error: 

 

ERROR Utils: Aborting task
java.lang.UnsatisfiedLinkError: /tmp/snappy-1.1.7-91d7ec9a-6e2d-47af-90ae-84c00d143b6d-libsnappyjava.so: Error loading shared library ld-linux-x86-64.so.2: No such file or directory

 

Looking into the executor pod, ld-linux-x86-64.so.2 is existed in /lib64 but not in /lib :

 

# ls -al /lib/ld*
-rwxr-xr-x 1 root root 584304 Mar 19 09:56 /lib/ld-musl-x86_64.so.1

# ls -al /lib64/ld* lrwxrwxrwx 1 root root 26 Jul 2 09:05 /lib64/ld-linux-x86-64.so.2 -> /lib/libc.musl-x86_64.so.1

After adding the gcompat in Dockerfile,  ld-linux-x86-64.so.2 exists in both lib and lib64 folders and no more error:

ls -al /lib/ld*
-rwxr-xr-x 1 root root 18512 Dec 24 2018 /lib/ld-linux-x86-64.so.2
-rwxr-xr-x 1 root root 584304 Mar 19 09:56 /lib/ld-musl-x86_64.so.1

ls -al /lib64/ld*
lrwxrwxrwx 1 root root 26 Jul 11 02:06 /lib64/ld-linux-x86-64.so.2 -> /lib/libc.musl-x86_64.so.1

 

 

 

Issue Links

• duplicates: SPARK-26995 Running Spark in Docker image with Alpine Linux 3.9.0 throws errors when using snappy
• links to: GitHub Pull Request #25112