[FLUME-580] Flume needs to be consistent with autodiscovery of Hadoop compression codecs

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.3
• Fix Version/s: 0.9.5
• Assignee: Jonathan Hsieh
• Reporter: Disabled imported user
• Created: 23/Mar/11 21:24
• Updated: 20/Sep/11 15:55
• Resolved: 20/Sep/11 15:55

Description

Currently, if flume notices that LZO is configured in core-site.xml it tries to load it. However, since it doesn't follow the same route that Hadoop does for locating the JAR and corresponding so files it fails to do so. Flume has to leverage the same logic that hadoop has embedded in bin/hadoop:
.....

1. setup 'java.library.path' for native-hadoop code if necessary
   if [ -d "${HADOOP_HOME}/build/native" -o -d "${HADOOP_HOME}/lib/native" -o -d "${HADOOP_HOME}/sbin" ]; then
   JAVA_PLATFORM=`CLASSPATH=${CLASSPATH} ${JAVA} -Xmx32m ${HADOOP_JAVA_PLATFORM_OPTS} org.apache.hadoop.util.PlatformName | sed -e "s/ /_/g"`
   ...
   so that it can transparently find LZO objects that it tries to autodiscover.

For the time being, however, the workaround seems to be to tell flume explicitly where to look for LZO via
adding the following to /usr/lib/flume/bin/flume-env.sh:

export JAVA_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/
export FLUME_CLASSPATH=/usr/lib/hadoop-0.20/lib/hadoop-lzo-20101122174751.20101122171345.552b3f9.jar