[BIGTOP-3535] Add zstd compression headers to add native lib support

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 3.0.0
• Component/s: hadoop
• Assignee: Xiao Yu
• Reporter: Xiao Yu
• Created: 15/Apr/21 15:42
• Updated: 21/Apr/21 00:33
• Resolved: 21/Apr/21 00:33

Description

Zstandard compression support was added to Hadoop 2.9.0 with HADOOP-13578 it would be great if we bundled support for it with native lib. As it stands now with Bigtop 1.5.0 we don't support it:

$ hadoop checknative -a
21/04/13 23:18:20 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
21/04/13 23:18:20 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
snappy:  true /usr/lib/hadoop/lib/native/libsnappy.so.1
zstd  :  false
lz4:     true revision:10301
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so
21/04/13 23:18:20 INFO util.ExitUtil: Exiting with status 1: ExitException

At least with Debian, after installing headers for zstd with apt-get install libzstd-dev and rebuilding the native lib with mvn package -Pdist,native -DskipTests -Drequire.zstd ... support appears to be enabled:

$ hadoop checknative -a
21/04/13 23:25:46 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
21/04/13 23:25:46 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
snappy:  true /usr/lib/hadoop/lib/native/libsnappy.so.1
zstd  :  true /usr/lib/x86_64-linux-gnu/libzstd.so.1
lz4:     true revision:10301
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so

Issue Links

• is duplicated by: BIGTOP-3509 Bundle ZStandard native library in Hadoop 3
• links to: GitHub PR