Flume
  1. Flume
  2. FLUME-383

E2E agent keeps dying when using batch and gzip decorators.

    Details

    • Type: Bug Bug
    • Status: Reopened
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v0.9.2
    • Fix Version/s: v0.9.5
    • Component/s: Node
    • Labels:
      None
    • Environment:

      CentOS 5.4
      Java build 1.6.0_13-b03

      Description

      When running in E2E using batch and gzip decorators, the agent dies.

      Configs:
      exec config 'xxx' 'syslogTcp(5140)' '{delay (500) => {batch(10000,30000) =>

      {gzip => autoE2EChain}

      }}'

      exec config 'xxx' 'autoCollectorSource' '{gunzip =>

      {unbatch => collectorSink("hdfs://dc1-had01-m001.dc01.revsci.net:9000/data/pixlogs","","60000")}

      }'

      Link to Flume user group thread
      https://groups.google.com/a/cloudera.org/group/flume-user/browse_thread/thread/c3195f16324f43c8#

      2010-12-06 15:29:57,669 ERROR
      com.cloudera.flume.handlers.batch.BatchingDecorator: IOException when
      ending batch!
      java.io.IOException: java.lang.NullPointerException
      at com.cloudera.flume.agent.durability.NaiveFileWALDeco
      $1.fireError(NaiveFileWALDeco.java:250)
      at
      com.cloudera.flume.core.connector.DirectDriver.fireError(DirectDriver.java:
      216)
      at com.cloudera.flume.core.connector.DirectDriver
      $PumperThread.run(DirectDriver.java:102)
      Caused by: java.lang.NullPointerException
      at
      com.cloudera.flume.handlers.thrift.ThriftEventSink.append(ThriftEventSink.java:
      76)
      at
      com.cloudera.flume.core.BackOffFailOverSink.append(BackOffFailOverSink.java:
      144)
      at
      com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:
      58)
      at
      com.cloudera.flume.handlers.debug.InsistentOpenDecorator.append(InsistentOpenDecorator.java:
      163)
      at
      com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:
      58)
      at
      com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:
      84)
      at
      com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:
      58)
      at com.cloudera.flume.agent.durability.NaiveFileWALDeco
      $AckChecksumRegisterer.append(NaiveFileWALDeco.java:99)
      at
      com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:
      58)
      at
      com.cloudera.flume.handlers.debug.LazyOpenDecorator.append(LazyOpenDecorator.java:
      70)
      at
      com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:
      58)
      at com.cloudera.flume.core.connector.DirectDriver
      $PumperThread.run(DirectDriver.java:93)

        Activity

        Hide
        Jonathan Hsieh added a comment -

        I'm pretty convinced that we are going to change batch and compression on rpc connections to be inside the sinks and sources instead of at the data flow level. This may be obviated when FLUME-559 gets implemented.

        Show
        Jonathan Hsieh added a comment - I'm pretty convinced that we are going to change batch and compression on rpc connections to be inside the sinks and sources instead of at the data flow level. This may be obviated when FLUME-559 gets implemented.

          People

          • Assignee:
            Unassigned
            Reporter:
            Disabled imported user
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development