[HADOOP-10409] Bzip2 error message isn't clear

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.3.0
• Component/s: io
• Assignee: Mohammad Islam
• Reporter: Travis Thompson
• Created: 14/Mar/14 23:28
• Updated: 19/Feb/15 19:11
• Resolved: 04/Apr/14 00:55

Description

If you compile hadoop without bzip2-devel installed (on RHEL), bzip2 doesn't get compiled into libhadoop, as is expected. This is not documented however and the error message thrown from hadoop checknative -a is not helpful.

[tthompso@eat1-hcl4060 bin]$ hadoop checknative -a
14/03/13 00:51:02 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
14/03/13 00:51:02 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /export/apps/hadoop/hadoop-2.3.0.li7-1-bin/lib/native/libhadoop.so.1.0.0
zlib:   true /lib64/libz.so.1
snappy: true /usr/lib64/libsnappy.so.1
lz4:    true revision:99
bzip2:  false 
14/03/13 00:51:02 INFO util.ExitUtil: Exiting with status 1

You can see that it wasn't compiled in here:

[mislam@eat1-hcl4060 ~]$ strings /export/apps/hadoop/latest/lib/native/libhadoop.so | grep initIDs
Java_org_apache_hadoop_io_compress_lz4_Lz4Compressor_initIDs
Java_org_apache_hadoop_io_compress_lz4_Lz4Decompressor_initIDs
Java_org_apache_hadoop_io_compress_snappy_SnappyCompressor_initIDs
Java_org_apache_hadoop_io_compress_snappy_SnappyDecompressor_initIDs
Java_org_apache_hadoop_io_compress_zlib_ZlibCompressor_initIDs
Java_org_apache_hadoop_io_compress_zlib_ZlibDecompressor_initIDs

After installing bzip2-devel and recompiling:

[tthompso@eat1-hcl4060 ~]$ hadoop checknative -a
14/03/14 23:00:08 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
14/03/14 23:00:08 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /export/apps/hadoop/hadoop-2.3.0.11-2-bin/lib/native/libhadoop.so.1.0.0
zlib:   true /lib64/libz.so.1
snappy: true /usr/lib64/libsnappy.so.1
lz4:    true revision:99
bzip2:  true /lib64/libbz2.so.1

tthompso@esv4-hcl261:~/hadoop-common(li-2.3.0⚡) » strings ./hadoop-common-project/hadoop-common/target/native/target/usr/local/lib/libhadoop.so |grep initIDs
Java_org_apache_hadoop_io_compress_lz4_Lz4Compressor_initIDs
Java_org_apache_hadoop_io_compress_lz4_Lz4Decompressor_initIDs
Java_org_apache_hadoop_io_compress_snappy_SnappyCompressor_initIDs
Java_org_apache_hadoop_io_compress_snappy_SnappyDecompressor_initIDs
Java_org_apache_hadoop_io_compress_zlib_ZlibCompressor_initIDs
Java_org_apache_hadoop_io_compress_zlib_ZlibDecompressor_initIDs
Java_org_apache_hadoop_io_compress_bzip2_Bzip2Compressor_initIDs
Java_org_apache_hadoop_io_compress_bzip2_Bzip2Decompressor_initIDs

The error message thrown should hint that perhaps libhadoop wasn't compiled with the bzip2 headers installed. It would also be nice if compile time dependencies were documented somewhere...

Issue Links

• is related to: HADOOP-8642 Document that io.native.lib.available only controls native bz2 and zlib compression codecs
• is related to: HADOOP-10452 BUILDING.txt needs to be updated