Minor Description
Zstandard compression support was added to Hadoop 2.9.0 with HADOOP-13578 it would be great if we bundled support for it with native lib. As it stands now with Bigtop 1.5.0 we don't support it:
$ hadoop checknative -a 21/04/13 23:18:20 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native 21/04/13 23:18:20 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library Native library checking: hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0 zlib: true /lib/x86_64-linux-gnu/libz.so.1 snappy: true /usr/lib/hadoop/lib/native/libsnappy.so.1 zstd : false lz4: true revision:10301 bzip2: true /lib/x86_64-linux-gnu/libbz2.so.1 openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so 21/04/13 23:18:20 INFO util.ExitUtil: Exiting with status 1: ExitException
At least with Debian, after installing headers for zstd with apt-get install libzstd-dev and rebuilding the native lib with mvn package -Pdist,native -DskipTests -Drequire.zstd ... support appears to be enabled:
$ hadoop checknative -a 21/04/13 23:25:46 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native 21/04/13 23:25:46 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library Native library checking: hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0 zlib: true /lib/x86_64-linux-gnu/libz.so.1 snappy: true /usr/lib/hadoop/lib/native/libsnappy.so.1 zstd : true /usr/lib/x86_64-linux-gnu/libzstd.so.1 lz4: true revision:10301 bzip2: true /lib/x86_64-linux-gnu/libbz2.so.1 openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so
Attachments
Issue Links
- is duplicated by
-
BIGTOP-3509 Bundle ZStandard native library in Hadoop 3
-
- Resolved
-
- links to
