Flume
  1. Flume
  2. FLUME-2370

'deflate' doesn't work in Flume avro sinc/source

    Details

    • Type: Documentation Documentation
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: v1.4.0
    • Fix Version/s: None
    • Component/s: Docs
    • Labels:
      None

      Description

      I was trying to setup a basic 2-tier Flume setup using the Avro source/sink to communicate between tiers. When I setup the system I used compression-type on the 1st tier's sink side like documented here (https://flume.apache.org/FlumeUserGuide.html#avro-sink)

      agent.sinks.snk1.compression-type = deflate

      And the same on the source side:
      agent.sources.src1.compression-type = deflate

      But I was getting this error:
      08 Apr 2014 00:15:07,606 WARN pool-7-thread-1 (org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.exceptionCaught:201) - Unexpected exception from downstream.
      org.jboss.netty.handler.codec.compression.CompressionException: decompression failure (-3): unknown compression method
      at org.jboss.netty.handler.codec.compression.ZlibUtil.exception(ZlibUtil.java:31)
      at org.jboss.netty.handler.codec.compression.ZlibUtil.fail(ZlibUtil.java:27)
      at org.jboss.netty.handler.codec.compression.ZlibDecoder.decode(ZlibDecoder.java:153)
      at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:67)
      at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
      at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
      at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
      at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
      at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
      at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:744)

      After some digging I found if I changed 'deflate' for 'gzip' it works.

      agent.sinks.snk1.compression-type = gzip

      agent.sources.src1.compression-type = gzip

      1. patch-2370.diff
        3 kB
        Cesar Delgado

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Unassigned
            Reporter:
            Cesar Delgado
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development