Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2370

'deflate' doesn't work in Flume avro sinc/source

    Details

    • Type: Documentation
    • Status: Resolved
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 1.4.0
    • Fix Version/s: None
    • Component/s: Docs
    • Labels:
      None

      Description

      I was trying to setup a basic 2-tier Flume setup using the Avro source/sink to communicate between tiers. When I setup the system I used compression-type on the 1st tier's sink side like documented here (https://flume.apache.org/FlumeUserGuide.html#avro-sink)

      agent.sinks.snk1.compression-type = deflate

      And the same on the source side:
      agent.sources.src1.compression-type = deflate

      But I was getting this error:
      08 Apr 2014 00:15:07,606 WARN pool-7-thread-1 (org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.exceptionCaught:201) - Unexpected exception from downstream.
      org.jboss.netty.handler.codec.compression.CompressionException: decompression failure (-3): unknown compression method
      at org.jboss.netty.handler.codec.compression.ZlibUtil.exception(ZlibUtil.java:31)
      at org.jboss.netty.handler.codec.compression.ZlibUtil.fail(ZlibUtil.java:27)
      at org.jboss.netty.handler.codec.compression.ZlibDecoder.decode(ZlibDecoder.java:153)
      at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:67)
      at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
      at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
      at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
      at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
      at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
      at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:744)

      After some digging I found if I changed 'deflate' for 'gzip' it works.

      agent.sinks.snk1.compression-type = gzip

      agent.sources.src1.compression-type = gzip

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              cdelgado@hortonworks.com Cesar Delgado
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: