Flume
  1. Flume
  2. FLUME-1093

HDFSEventSink gives no indication of failure when Hadoop jars aren't in the classpath

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Not A Problem
    • Affects Version/s: v1.2.0
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Environment:

      CentOS 5.7 64-bit

      Description

      If HDFS sink starts up, but the hadoop jars aren't in the classpath, then the HDFS sink doesn't fail-fast with an error. I think we should fail the validation of the config file if it specifies an HDFS Sink when the hadoop jars aren't in the classpath.

        Issue Links

          Activity

          Hide
          Hari Shreedharan added a comment -

          Great! I will close this one.

          Show
          Hari Shreedharan added a comment - Great! I will close this one.
          Hide
          Ashish Paliwal added a comment -

          This issue has been fixed as part of FLUME-1630. Tried reproducing it and a proper error is logged, when dependent classes are not found.

          Hari Shreedharan We can mark this issue as resolved.

          Show
          Ashish Paliwal added a comment - This issue has been fixed as part of FLUME-1630 . Tried reproducing it and a proper error is logged, when dependent classes are not found. Hari Shreedharan We can mark this issue as resolved.
          Hide
          Hari Shreedharan added a comment -

          I had posted an attachment on FLUME-1246 a few days back, which does the same thing. It is waiting on review. We really need to catch a NoClassDefFoundError, I believe.

          Show
          Hari Shreedharan added a comment - I had posted an attachment on FLUME-1246 a few days back, which does the same thing. It is waiting on review. We really need to catch a NoClassDefFoundError, I believe.
          Hide
          Mubarak Seyed added a comment -

          The attached file is a patch. When sink class is not in classpath then we catch the exception and log it as

          "Unable to load sink type: " + type
                    + ", class: " + sinkClassName, ex);
          
          Show
          Mubarak Seyed added a comment - The attached file is a patch. When sink class is not in classpath then we catch the exception and log it as "Unable to load sink type: " + type + ", class: " + sinkClassName, ex);
          Hide
          Mubarak Seyed added a comment -

          Taking a look

          Show
          Mubarak Seyed added a comment - Taking a look
          Hide
          Leslin (Hong Xiang Lin) added a comment -

          There are two ways to solve this problem:
          1. Pre-check to make sure that all required jars are exist. This will be easy.
          2. Second(maybe more reasonable), there should be exception thrown up when jars unavailable. But why no exception in this case? Anyone has suggestion here?
          Below is source code in use.
          Class<? extends Sink> sinkClass = null;
          try

          { sinkClass = (Class<? extends Sink>) Class.forName(sinkClassName); }

          catch (Exception ex)

          { throw new FlumeException("Unable to load sink type: " + type + ", class: " + sinkClassName, ex); }

          3. Below is hang screen log for your reference.
          leslin@ubtServer:~$ flume-ng agent -n agent -f /usr/lib/flume-ng/conf/flume-conf.properties
          Warning: No configuration directory set! Use --conf <dir> to override.
          Warning: JAVA_HOME not set!
          Info: Including Hadoop libraries found via (/usr/bin/hadoop) for HDFS access
          Info: Including HBASE libraries found via (/usr/bin/hbase) for HBASE access
          Info: Excluding /usr/lib/hbase/bin/../lib/slf4j-api-1.5.8.jar from classpath
          Info: Excluding /usr/lib/hbase/bin/../lib/slf4j-log4j12-1.5.8.jar from classpath
          Info: Excluding /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar from classpath
          Info: Excluding /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar from classpath
          Info: Excluding /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar from classpath
          Info: Excluding /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar from classpath
          -------------------------------------
          FLUME_CLASSPATH: /usr/lib/flume-ng/lib/*:/usr/lib/hbase/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/lib/hbase/bin/..:/usr/lib/hbase/bin/../hbase-0.90.6-cdh3u4.jar:/usr/lib/hbase/bin/../hbase-0.90.6-cdh3u4-tests.jar:/usr/lib/hbase/bin/../lib/activation-1.1.jar:/usr/lib/hbase/bin/../lib/asm-3.1.jar:/usr/lib/hbase/bin/../lib/avro-1.5.4.jar:/usr/lib/hbase/bin/../lib/avro-ipc-1.5.4.jar:/usr/lib/hbase/bin/../lib/commons-cli-1.2.jar:/usr/lib/hbase/bin/../lib/commons-codec-1.4.jar:/usr/lib/hbase/bin/../lib/commons-el-1.0.jar:/usr/lib/hbase/bin/../lib/commons-httpclient-3.1.jar:/usr/lib/hbase/bin/../lib/commons-lang-2.5.jar:/usr/lib/hbase/bin/../lib/commons-logging-1.1.1.jar:/usr/lib/hbase/bin/../lib/commons-net-1.4.1.jar:/usr/lib/hbase/bin/../lib/core-3.1.1.jar:/usr/lib/hbase/bin/../lib/guava-r06.jar:/usr/lib/hbase/bin/../lib/guava-r09-jarjar.jar:/usr/lib/hbase/bin/../lib/hadoop-core.jar:/usr/lib/hbase/bin/../lib/jackson-core-asl-1.5.2.jar:/usr/lib/hbase/bin/../lib/jackson-jaxrs-1.5.5.jar:/usr/lib/hbase/bin/../lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hbase/bin/../lib/jackson-xc-1.5.5.jar:/usr/lib/hbase/bin/../lib/jamon-runtime-2.3.1.jar:/usr/lib/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/usr/lib/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/usr/lib/hbase/bin/../lib/jaxb-api-2.1.jar:/usr/lib/hbase/bin/../lib/jaxb-impl-2.1.12.jar:/usr/lib/hbase/bin/../lib/jersey-core-1.4.jar:/usr/lib/hbase/bin/../lib/jersey-json-1.4.jar:/usr/lib/hbase/bin/../lib/jersey-server-1.4.jar:/usr/lib/hbase/bin/../lib/jettison-1.1.jar:/usr/lib/hbase/bin/../lib/jetty-6.1.26.jar:/usr/lib/hbase/bin/../lib/jetty-util-6.1.26.jar:/usr/lib/hbase/bin/../lib/jruby-complete-1.6.0.jar:/usr/lib/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/usr/lib/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/lib/hbase/bin/../lib/jsp-api-2.1.jar:/usr/lib/hbase/bin/../lib/jsr311-api-1.1.1.jar:/usr/lib/hbase/bin/../lib/log4j-1.2.16.jar:/usr/lib/hbase/bin/../lib/netty-3.2.4.Final.jar:/usr/lib/hbase/bin/../lib/protobuf-java-2.3.0.jar:/usr/lib/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hbase/bin/../lib/servlet-api-2.5.jar:/usr/lib/hbase/bin/../lib/snappy-java-1.0.3.2.jar:/usr/lib/hbase/bin/../lib/stax-api-1.0.1.jar:/usr/lib/hbase/bin/../lib/thrift-0.2.0.jar:/usr/lib/hbase/bin/../lib/velocity-1.5.jar:/usr/lib/hbase/bin/../lib/xmlenc-0.52.jar:/usr/lib/hbase/bin/../lib/zookeeper.jar:/etc/zookeeper:/etc/hadoop-0.20/conf:/usr/lib/hadoop-0.20/bin:/usr/lib/hadoop-0.20/build.xml:/usr/lib/hadoop-0.20/CHANGES.txt:/usr/lib/hadoop-0.20/conf:/usr/lib/hadoop-0.20/contrib:/usr/lib/hadoop-0.20/example-confs:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-ant.jar:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-core.jar:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-examples.jar:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-test.jar:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-tools.jar:/usr/lib/hadoop-0.20/hadoop-ant-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/hadoop-ant.jar:/usr/lib/hadoop-0.20/hadoop-core.jar:/usr/lib/hadoop-0.20/hadoop-examples-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/hadoop-examples.jar:/usr/lib/hadoop-0.20/hadoop-jar-backup:/usr/lib/hadoop-0.20/hadoop-test-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/hadoop-test.jar:/usr/lib/hadoop-0.20/hadoop-tools-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/hadoop-tools.jar:/usr/lib/hadoop-0.20/ivy:/usr/lib/hadoop-0.20/ivy.xml:/usr/lib/hadoop-0.20/lib:/usr/lib/hadoop-0.20/logs:/usr/lib/hadoop-0.20/NOTICE.txt:/usr/lib/hadoop-0.20/pids:/usr/lib/hadoop-0.20/README.txt:/usr/lib/hadoop-0.20/webapps:/usr/lib/hadoop-0.20/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20/lib/commons-lang-2.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/guava-r09-jarjar.jar:/usr/lib/hadoop-0.20/lib/hadoop-fairscheduler-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jackson-core-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jdiff:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-servlet-tester-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1:/usr/lib/hadoop-0.20/lib/junit-4.5.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop-0.20/lib/native:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/zookeeper/bin:/usr/lib/zookeeper/conf:/usr/lib/zookeeper/lib:/usr/lib/zookeeper/zookeeper-3.3.5-cdh3u4.jar:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/conf:/usr/lib/hadoop-0.20/hadoop-core.jar:/usr/lib/hadoop-0.20/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20/lib/commons-lang-2.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/guava-r09-jarjar.jar:/usr/lib/hadoop-0.20/lib/hadoop-fairscheduler-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jackson-core-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-servlet-tester-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20/lib/junit-4.5.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/conf
          FLUME_JAVA_LIBRARY_PATH: :/usr/lib/hadoop-0.20/lib/native/:/usr/lib/hbase/bin/../lib/native/
          FLUME_APPLICATION_CLASS: org.apache.flume.node.Application
          -------------------------------------
          + exec /usr/bin/java -Xmx20m -cp '/usr/lib/flume-ng/lib/*:/usr/lib/hbase/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/lib/hbase/bin/..:/usr/lib/hbase/bin/../hbase-0.90.6-cdh3u4.jar:/usr/lib/hbase/bin/../hbase-0.90.6-cdh3u4-tests.jar:/usr/lib/hbase/bin/../lib/activation-1.1.jar:/usr/lib/hbase/bin/../lib/asm-3.1.jar:/usr/lib/hbase/bin/../lib/avro-1.5.4.jar:/usr/lib/hbase/bin/../lib/avro-ipc-1.5.4.jar:/usr/lib/hbase/bin/../lib/commons-cli-1.2.jar:/usr/lib/hbase/bin/../lib/commons-codec-1.4.jar:/usr/lib/hbase/bin/../lib/commons-el-1.0.jar:/usr/lib/hbase/bin/../lib/commons-httpclient-3.1.jar:/usr/lib/hbase/bin/../lib/commons-lang-2.5.jar:/usr/lib/hbase/bin/../lib/commons-logging-1.1.1.jar:/usr/lib/hbase/bin/../lib/commons-net-1.4.1.jar:/usr/lib/hbase/bin/../lib/core-3.1.1.jar:/usr/lib/hbase/bin/../lib/guava-r06.jar:/usr/lib/hbase/bin/../lib/guava-r09-jarjar.jar:/usr/lib/hbase/bin/../lib/hadoop-core.jar:/usr/lib/hbase/bin/../lib/jackson-core-asl-1.5.2.jar:/usr/lib/hbase/bin/../lib/jackson-jaxrs-1.5.5.jar:/usr/lib/hbase/bin/../lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hbase/bin/../lib/jackson-xc-1.5.5.jar:/usr/lib/hbase/bin/../lib/jamon-runtime-2.3.1.jar:/usr/lib/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/usr/lib/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/usr/lib/hbase/bin/../lib/jaxb-api-2.1.jar:/usr/lib/hbase/bin/../lib/jaxb-impl-2.1.12.jar:/usr/lib/hbase/bin/../lib/jersey-core-1.4.jar:/usr/lib/hbase/bin/../lib/jersey-json-1.4.jar:/usr/lib/hbase/bin/../lib/jersey-server-1.4.jar:/usr/lib/hbase/bin/../lib/jettison-1.1.jar:/usr/lib/hbase/bin/../lib/jetty-6.1.26.jar:/usr/lib/hbase/bin/../lib/jetty-util-6.1.26.jar:/usr/lib/hbase/bin/../lib/jruby-complete-1.6.0.jar:/usr/lib/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/usr/lib/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/lib/hbase/bin/../lib/jsp-api-2.1.jar:/usr/lib/hbase/bin/../lib/jsr311-api-1.1.1.jar:/usr/lib/hbase/bin/../lib/log4j-1.2.16.jar:/usr/lib/hbase/bin/../lib/netty-3.2.4.Final.jar:/usr/lib/hbase/bin/../lib/protobuf-java-2.3.0.jar:/usr/lib/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hbase/bin/../lib/servlet-api-2.5.jar:/usr/lib/hbase/bin/../lib/snappy-java-1.0.3.2.jar:/usr/lib/hbase/bin/../lib/stax-api-1.0.1.jar:/usr/lib/hbase/bin/../lib/thrift-0.2.0.jar:/usr/lib/hbase/bin/../lib/velocity-1.5.jar:/usr/lib/hbase/bin/../lib/xmlenc-0.52.jar:/usr/lib/hbase/bin/../lib/zookeeper.jar:/etc/zookeeper:/etc/hadoop-0.20/conf:/usr/lib/hadoop-0.20/bin:/usr/lib/hadoop-0.20/build.xml:/usr/lib/hadoop-0.20/CHANGES.txt:/usr/lib/hadoop-0.20/conf:/usr/lib/hadoop-0.20/contrib:/usr/lib/hadoop-0.20/example-confs:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-ant.jar:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-core.jar:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-examples.jar:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-test.jar:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-tools.jar:/usr/lib/hadoop-0.20/hadoop-ant-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/hadoop-ant.jar:/usr/lib/hadoop-0.20/hadoop-core.jar:/usr/lib/hadoop-0.20/hadoop-examples-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/hadoop-examples.jar:/usr/lib/hadoop-0.20/hadoop-jar-backup:/usr/lib/hadoop-0.20/hadoop-test-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/hadoop-test.jar:/usr/lib/hadoop-0.20/hadoop-tools-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/hadoop-tools.jar:/usr/lib/hadoop-0.20/ivy:/usr/lib/hadoop-0.20/ivy.xml:/usr/lib/hadoop-0.20/lib:/usr/lib/hadoop-0.20/logs:/usr/lib/hadoop-0.20/NOTICE.txt:/usr/lib/hadoop-0.20/pids:/usr/lib/hadoop-0.20/README.txt:/usr/lib/hadoop-0.20/webapps:/usr/lib/hadoop-0.20/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20/lib/commons-lang-2.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/guava-r09-jarjar.jar:/usr/lib/hadoop-0.20/lib/hadoop-fairscheduler-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jackson-core-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jdiff:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-servlet-tester-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1:/usr/lib/hadoop-0.20/lib/junit-4.5.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop-0.20/lib/native:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/zookeeper/bin:/usr/lib/zookeeper/conf:/usr/lib/zookeeper/lib:/usr/lib/zookeeper/zookeeper-3.3.5-cdh3u4.jar:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/conf:/usr/lib/hadoop-0.20/hadoop-core.jar:/usr/lib/hadoop-0.20/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20/lib/commons-lang-2.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/guava-r09-jarjar.jar:/usr/lib/hadoop-0.20/lib/hadoop-fairscheduler-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jackson-core-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-servlet-tester-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20/lib/junit-4.5.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/conf' -Djava.library.path=:/usr/lib/hadoop-0.20/lib/native/:/usr/lib/hbase/bin/../lib/native/ org.apache.flume.node.Application -n agent -f /usr/lib/flume-ng/conf/flume-conf.properties
          12/06/15 22:49:20 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 1
          12/06/15 22:49:20 INFO node.FlumeNode: Flume node starting - agent
          12/06/15 22:49:20 INFO nodemanager.DefaultLogicalNodeManager: Node manager starting
          12/06/15 22:49:20 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 11
          12/06/15 22:49:20 INFO properties.PropertiesFileConfigurationProvider: Configuration provider starting
          12/06/15 22:49:20 INFO properties.PropertiesFileConfigurationProvider: Reloading configuration file:/usr/lib/flume-ng/conf/flume-conf.properties
          12/06/15 22:49:20 INFO conf.FlumeConfiguration: Added sinks: k1 Agent: agent
          12/06/15 22:49:20 INFO conf.FlumeConfiguration: Processing:k1
          12/06/15 22:49:20 INFO conf.FlumeConfiguration: Processing:k1
          12/06/15 22:49:20 INFO conf.FlumeConfiguration: Processing:k1
          12/06/15 22:49:20 INFO conf.FlumeConfiguration: Processing:k1
          12/06/15 22:49:20 INFO conf.FlumeConfiguration: Processing:k1
          12/06/15 22:49:20 INFO conf.FlumeConfiguration: Processing:k1
          12/06/15 22:49:20 INFO conf.FlumeConfiguration: Processing:k1
          12/06/15 22:49:20 INFO conf.FlumeConfiguration: Processing:k1
          12/06/15 22:49:20 INFO conf.FlumeConfiguration: Processing:k1
          12/06/15 22:49:20 INFO conf.FlumeConfiguration: Processing:k1
          12/06/15 22:49:20 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent]
          12/06/15 22:49:20 INFO properties.PropertiesFileConfigurationProvider: Creating channels
          12/06/15 22:49:20 INFO properties.PropertiesFileConfigurationProvider: created channel c1
          12/06/15 22:49:20 INFO sink.DefaultSinkFactory: Creating instance of sink k1 typeHDFS

          Show
          Leslin (Hong Xiang Lin) added a comment - There are two ways to solve this problem: 1. Pre-check to make sure that all required jars are exist. This will be easy. 2. Second(maybe more reasonable), there should be exception thrown up when jars unavailable. But why no exception in this case? Anyone has suggestion here? Below is source code in use. Class<? extends Sink> sinkClass = null; try { sinkClass = (Class<? extends Sink>) Class.forName(sinkClassName); } catch (Exception ex) { throw new FlumeException("Unable to load sink type: " + type + ", class: " + sinkClassName, ex); } 3. Below is hang screen log for your reference. leslin@ubtServer:~$ flume-ng agent -n agent -f /usr/lib/flume-ng/conf/flume-conf.properties Warning: No configuration directory set! Use --conf <dir> to override. Warning: JAVA_HOME not set! Info: Including Hadoop libraries found via (/usr/bin/hadoop) for HDFS access Info: Including HBASE libraries found via (/usr/bin/hbase) for HBASE access Info: Excluding /usr/lib/hbase/bin/../lib/slf4j-api-1.5.8.jar from classpath Info: Excluding /usr/lib/hbase/bin/../lib/slf4j-log4j12-1.5.8.jar from classpath Info: Excluding /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar from classpath Info: Excluding /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar from classpath Info: Excluding /usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar from classpath Info: Excluding /usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar from classpath ------------------------------------- FLUME_CLASSPATH: /usr/lib/flume-ng/lib/*:/usr/lib/hbase/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/lib/hbase/bin/..:/usr/lib/hbase/bin/../hbase-0.90.6-cdh3u4.jar:/usr/lib/hbase/bin/../hbase-0.90.6-cdh3u4-tests.jar:/usr/lib/hbase/bin/../lib/activation-1.1.jar:/usr/lib/hbase/bin/../lib/asm-3.1.jar:/usr/lib/hbase/bin/../lib/avro-1.5.4.jar:/usr/lib/hbase/bin/../lib/avro-ipc-1.5.4.jar:/usr/lib/hbase/bin/../lib/commons-cli-1.2.jar:/usr/lib/hbase/bin/../lib/commons-codec-1.4.jar:/usr/lib/hbase/bin/../lib/commons-el-1.0.jar:/usr/lib/hbase/bin/../lib/commons-httpclient-3.1.jar:/usr/lib/hbase/bin/../lib/commons-lang-2.5.jar:/usr/lib/hbase/bin/../lib/commons-logging-1.1.1.jar:/usr/lib/hbase/bin/../lib/commons-net-1.4.1.jar:/usr/lib/hbase/bin/../lib/core-3.1.1.jar:/usr/lib/hbase/bin/../lib/guava-r06.jar:/usr/lib/hbase/bin/../lib/guava-r09-jarjar.jar:/usr/lib/hbase/bin/../lib/hadoop-core.jar:/usr/lib/hbase/bin/../lib/jackson-core-asl-1.5.2.jar:/usr/lib/hbase/bin/../lib/jackson-jaxrs-1.5.5.jar:/usr/lib/hbase/bin/../lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hbase/bin/../lib/jackson-xc-1.5.5.jar:/usr/lib/hbase/bin/../lib/jamon-runtime-2.3.1.jar:/usr/lib/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/usr/lib/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/usr/lib/hbase/bin/../lib/jaxb-api-2.1.jar:/usr/lib/hbase/bin/../lib/jaxb-impl-2.1.12.jar:/usr/lib/hbase/bin/../lib/jersey-core-1.4.jar:/usr/lib/hbase/bin/../lib/jersey-json-1.4.jar:/usr/lib/hbase/bin/../lib/jersey-server-1.4.jar:/usr/lib/hbase/bin/../lib/jettison-1.1.jar:/usr/lib/hbase/bin/../lib/jetty-6.1.26.jar:/usr/lib/hbase/bin/../lib/jetty-util-6.1.26.jar:/usr/lib/hbase/bin/../lib/jruby-complete-1.6.0.jar:/usr/lib/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/usr/lib/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/lib/hbase/bin/../lib/jsp-api-2.1.jar:/usr/lib/hbase/bin/../lib/jsr311-api-1.1.1.jar:/usr/lib/hbase/bin/../lib/log4j-1.2.16.jar:/usr/lib/hbase/bin/../lib/netty-3.2.4.Final.jar:/usr/lib/hbase/bin/../lib/protobuf-java-2.3.0.jar:/usr/lib/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hbase/bin/../lib/servlet-api-2.5.jar:/usr/lib/hbase/bin/../lib/snappy-java-1.0.3.2.jar:/usr/lib/hbase/bin/../lib/stax-api-1.0.1.jar:/usr/lib/hbase/bin/../lib/thrift-0.2.0.jar:/usr/lib/hbase/bin/../lib/velocity-1.5.jar:/usr/lib/hbase/bin/../lib/xmlenc-0.52.jar:/usr/lib/hbase/bin/../lib/zookeeper.jar:/etc/zookeeper:/etc/hadoop-0.20/conf:/usr/lib/hadoop-0.20/bin:/usr/lib/hadoop-0.20/build.xml:/usr/lib/hadoop-0.20/CHANGES.txt:/usr/lib/hadoop-0.20/conf:/usr/lib/hadoop-0.20/contrib:/usr/lib/hadoop-0.20/example-confs:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-ant.jar:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-core.jar:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-examples.jar:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-test.jar:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-tools.jar:/usr/lib/hadoop-0.20/hadoop-ant-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/hadoop-ant.jar:/usr/lib/hadoop-0.20/hadoop-core.jar:/usr/lib/hadoop-0.20/hadoop-examples-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/hadoop-examples.jar:/usr/lib/hadoop-0.20/hadoop-jar-backup:/usr/lib/hadoop-0.20/hadoop-test-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/hadoop-test.jar:/usr/lib/hadoop-0.20/hadoop-tools-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/hadoop-tools.jar:/usr/lib/hadoop-0.20/ivy:/usr/lib/hadoop-0.20/ivy.xml:/usr/lib/hadoop-0.20/lib:/usr/lib/hadoop-0.20/logs:/usr/lib/hadoop-0.20/NOTICE.txt:/usr/lib/hadoop-0.20/pids:/usr/lib/hadoop-0.20/README.txt:/usr/lib/hadoop-0.20/webapps:/usr/lib/hadoop-0.20/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20/lib/commons-lang-2.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/guava-r09-jarjar.jar:/usr/lib/hadoop-0.20/lib/hadoop-fairscheduler-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jackson-core-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jdiff:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-servlet-tester-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1:/usr/lib/hadoop-0.20/lib/junit-4.5.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop-0.20/lib/native:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/zookeeper/bin:/usr/lib/zookeeper/conf:/usr/lib/zookeeper/lib:/usr/lib/zookeeper/zookeeper-3.3.5-cdh3u4.jar:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/conf:/usr/lib/hadoop-0.20/hadoop-core.jar:/usr/lib/hadoop-0.20/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20/lib/commons-lang-2.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/guava-r09-jarjar.jar:/usr/lib/hadoop-0.20/lib/hadoop-fairscheduler-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jackson-core-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-servlet-tester-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20/lib/junit-4.5.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/conf FLUME_JAVA_LIBRARY_PATH: :/usr/lib/hadoop-0.20/lib/native/:/usr/lib/hbase/bin/../lib/native/ FLUME_APPLICATION_CLASS: org.apache.flume.node.Application ------------------------------------- + exec /usr/bin/java -Xmx20m -cp '/usr/lib/flume-ng/lib/*:/usr/lib/hbase/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/lib/hbase/bin/..:/usr/lib/hbase/bin/../hbase-0.90.6-cdh3u4.jar:/usr/lib/hbase/bin/../hbase-0.90.6-cdh3u4-tests.jar:/usr/lib/hbase/bin/../lib/activation-1.1.jar:/usr/lib/hbase/bin/../lib/asm-3.1.jar:/usr/lib/hbase/bin/../lib/avro-1.5.4.jar:/usr/lib/hbase/bin/../lib/avro-ipc-1.5.4.jar:/usr/lib/hbase/bin/../lib/commons-cli-1.2.jar:/usr/lib/hbase/bin/../lib/commons-codec-1.4.jar:/usr/lib/hbase/bin/../lib/commons-el-1.0.jar:/usr/lib/hbase/bin/../lib/commons-httpclient-3.1.jar:/usr/lib/hbase/bin/../lib/commons-lang-2.5.jar:/usr/lib/hbase/bin/../lib/commons-logging-1.1.1.jar:/usr/lib/hbase/bin/../lib/commons-net-1.4.1.jar:/usr/lib/hbase/bin/../lib/core-3.1.1.jar:/usr/lib/hbase/bin/../lib/guava-r06.jar:/usr/lib/hbase/bin/../lib/guava-r09-jarjar.jar:/usr/lib/hbase/bin/../lib/hadoop-core.jar:/usr/lib/hbase/bin/../lib/jackson-core-asl-1.5.2.jar:/usr/lib/hbase/bin/../lib/jackson-jaxrs-1.5.5.jar:/usr/lib/hbase/bin/../lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hbase/bin/../lib/jackson-xc-1.5.5.jar:/usr/lib/hbase/bin/../lib/jamon-runtime-2.3.1.jar:/usr/lib/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/usr/lib/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/usr/lib/hbase/bin/../lib/jaxb-api-2.1.jar:/usr/lib/hbase/bin/../lib/jaxb-impl-2.1.12.jar:/usr/lib/hbase/bin/../lib/jersey-core-1.4.jar:/usr/lib/hbase/bin/../lib/jersey-json-1.4.jar:/usr/lib/hbase/bin/../lib/jersey-server-1.4.jar:/usr/lib/hbase/bin/../lib/jettison-1.1.jar:/usr/lib/hbase/bin/../lib/jetty-6.1.26.jar:/usr/lib/hbase/bin/../lib/jetty-util-6.1.26.jar:/usr/lib/hbase/bin/../lib/jruby-complete-1.6.0.jar:/usr/lib/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/usr/lib/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/lib/hbase/bin/../lib/jsp-api-2.1.jar:/usr/lib/hbase/bin/../lib/jsr311-api-1.1.1.jar:/usr/lib/hbase/bin/../lib/log4j-1.2.16.jar:/usr/lib/hbase/bin/../lib/netty-3.2.4.Final.jar:/usr/lib/hbase/bin/../lib/protobuf-java-2.3.0.jar:/usr/lib/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hbase/bin/../lib/servlet-api-2.5.jar:/usr/lib/hbase/bin/../lib/snappy-java-1.0.3.2.jar:/usr/lib/hbase/bin/../lib/stax-api-1.0.1.jar:/usr/lib/hbase/bin/../lib/thrift-0.2.0.jar:/usr/lib/hbase/bin/../lib/velocity-1.5.jar:/usr/lib/hbase/bin/../lib/xmlenc-0.52.jar:/usr/lib/hbase/bin/../lib/zookeeper.jar:/etc/zookeeper:/etc/hadoop-0.20/conf:/usr/lib/hadoop-0.20/bin:/usr/lib/hadoop-0.20/build.xml:/usr/lib/hadoop-0.20/CHANGES.txt:/usr/lib/hadoop-0.20/conf:/usr/lib/hadoop-0.20/contrib:/usr/lib/hadoop-0.20/example-confs:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-ant.jar:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-core.jar:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-examples.jar:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-test.jar:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-tools.jar:/usr/lib/hadoop-0.20/hadoop-ant-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/hadoop-ant.jar:/usr/lib/hadoop-0.20/hadoop-core.jar:/usr/lib/hadoop-0.20/hadoop-examples-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/hadoop-examples.jar:/usr/lib/hadoop-0.20/hadoop-jar-backup:/usr/lib/hadoop-0.20/hadoop-test-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/hadoop-test.jar:/usr/lib/hadoop-0.20/hadoop-tools-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/hadoop-tools.jar:/usr/lib/hadoop-0.20/ivy:/usr/lib/hadoop-0.20/ivy.xml:/usr/lib/hadoop-0.20/lib:/usr/lib/hadoop-0.20/logs:/usr/lib/hadoop-0.20/NOTICE.txt:/usr/lib/hadoop-0.20/pids:/usr/lib/hadoop-0.20/README.txt:/usr/lib/hadoop-0.20/webapps:/usr/lib/hadoop-0.20/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20/lib/commons-lang-2.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/guava-r09-jarjar.jar:/usr/lib/hadoop-0.20/lib/hadoop-fairscheduler-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jackson-core-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jdiff:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-servlet-tester-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1:/usr/lib/hadoop-0.20/lib/junit-4.5.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop-0.20/lib/native:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/zookeeper/bin:/usr/lib/zookeeper/conf:/usr/lib/zookeeper/lib:/usr/lib/zookeeper/zookeeper-3.3.5-cdh3u4.jar:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/conf:/usr/lib/hadoop-0.20/hadoop-core.jar:/usr/lib/hadoop-0.20/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20/lib/commons-lang-2.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/guava-r09-jarjar.jar:/usr/lib/hadoop-0.20/lib/hadoop-fairscheduler-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jackson-core-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-servlet-tester-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20/lib/junit-4.5.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/conf' -Djava.library.path=:/usr/lib/hadoop-0.20/lib/native/:/usr/lib/hbase/bin/../lib/native/ org.apache.flume.node.Application -n agent -f /usr/lib/flume-ng/conf/flume-conf.properties 12/06/15 22:49:20 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 1 12/06/15 22:49:20 INFO node.FlumeNode: Flume node starting - agent 12/06/15 22:49:20 INFO nodemanager.DefaultLogicalNodeManager: Node manager starting 12/06/15 22:49:20 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 11 12/06/15 22:49:20 INFO properties.PropertiesFileConfigurationProvider: Configuration provider starting 12/06/15 22:49:20 INFO properties.PropertiesFileConfigurationProvider: Reloading configuration file:/usr/lib/flume-ng/conf/flume-conf.properties 12/06/15 22:49:20 INFO conf.FlumeConfiguration: Added sinks: k1 Agent: agent 12/06/15 22:49:20 INFO conf.FlumeConfiguration: Processing:k1 12/06/15 22:49:20 INFO conf.FlumeConfiguration: Processing:k1 12/06/15 22:49:20 INFO conf.FlumeConfiguration: Processing:k1 12/06/15 22:49:20 INFO conf.FlumeConfiguration: Processing:k1 12/06/15 22:49:20 INFO conf.FlumeConfiguration: Processing:k1 12/06/15 22:49:20 INFO conf.FlumeConfiguration: Processing:k1 12/06/15 22:49:20 INFO conf.FlumeConfiguration: Processing:k1 12/06/15 22:49:20 INFO conf.FlumeConfiguration: Processing:k1 12/06/15 22:49:20 INFO conf.FlumeConfiguration: Processing:k1 12/06/15 22:49:20 INFO conf.FlumeConfiguration: Processing:k1 12/06/15 22:49:20 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent] 12/06/15 22:49:20 INFO properties.PropertiesFileConfigurationProvider: Creating channels 12/06/15 22:49:20 INFO properties.PropertiesFileConfigurationProvider: created channel c1 12/06/15 22:49:20 INFO sink.DefaultSinkFactory: Creating instance of sink k1 typeHDFS
          Hide
          NO NAME added a comment -

          I had this same problem with the HBase sink and it was extremely frustrating. It's tough because:

          1) There is no error message - flume just hangs.
          2) JStack doesn't give any helpful output.

          Perhaps flume is eating an error here internally? It would be great if someone could pick this up - it is a major blocker to smooth deployment.

          Show
          NO NAME added a comment - I had this same problem with the HBase sink and it was extremely frustrating. It's tough because: 1) There is no error message - flume just hangs. 2) JStack doesn't give any helpful output. Perhaps flume is eating an error here internally? It would be great if someone could pick this up - it is a major blocker to smooth deployment.
          Hide
          Mike Percy added a comment -

          not looking at this at the moment, unassigning from myself...

          Show
          Mike Percy added a comment - not looking at this at the moment, unassigning from myself...
          Hide
          Brock Noland added a comment -

          Marking my JIRA a dup.

          Show
          Brock Noland added a comment - Marking my JIRA a dup.

            People

            • Assignee:
              Unassigned
              Reporter:
              Will McQueen
            • Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development