Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-580

Flume needs to be consistent with autodiscovery of Hadoop compression codecs

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.3
    • Fix Version/s: 0.9.5
    • Component/s: None
    • Labels:
      None

      Description

      Currently, if flume notices that LZO is configured in core-site.xml it tries to load it. However, since it doesn't follow the same route that Hadoop does for locating the JAR and corresponding so files it fails to do so. Flume has to leverage the same logic that hadoop has embedded in bin/hadoop:
      .....

      1. setup 'java.library.path' for native-hadoop code if necessary
        if [ -d "${HADOOP_HOME}/build/native" -o -d "${HADOOP_HOME}/lib/native" -o -d "${HADOOP_HOME}/sbin" ]; then
        JAVA_PLATFORM=`CLASSPATH=${CLASSPATH} ${JAVA} -Xmx32m ${HADOOP_JAVA_PLATFORM_OPTS} org.apache.hadoop.util.PlatformName | sed -e "s/ /_/g"`
        ...
        so that it can transparently find LZO objects that it tries to autodiscover.

      For the time being, however, the workaround seems to be to tell flume explicitly where to look for LZO via
      adding the following to /usr/lib/flume/bin/flume-env.sh:

      export JAVA_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/
      export FLUME_CLASSPATH=/usr/lib/hadoop-0.20/lib/hadoop-lzo-20101122174751.20101122171345.552b3f9.jar

        Attachments

          Activity

            People

            • Assignee:
              jmhsieh Jonathan Hsieh
              Reporter:
              flume_rvs Disabled imported user
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: