Flume
  1. Flume
  2. FLUME-2370

'deflate' doesn't work in Flume avro sinc/source

    Details

    • Type: Documentation Documentation
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: v1.4.0
    • Fix Version/s: None
    • Component/s: Docs
    • Labels:
      None

      Description

      I was trying to setup a basic 2-tier Flume setup using the Avro source/sink to communicate between tiers. When I setup the system I used compression-type on the 1st tier's sink side like documented here (https://flume.apache.org/FlumeUserGuide.html#avro-sink)

      agent.sinks.snk1.compression-type = deflate

      And the same on the source side:
      agent.sources.src1.compression-type = deflate

      But I was getting this error:
      08 Apr 2014 00:15:07,606 WARN pool-7-thread-1 (org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.exceptionCaught:201) - Unexpected exception from downstream.
      org.jboss.netty.handler.codec.compression.CompressionException: decompression failure (-3): unknown compression method
      at org.jboss.netty.handler.codec.compression.ZlibUtil.exception(ZlibUtil.java:31)
      at org.jboss.netty.handler.codec.compression.ZlibUtil.fail(ZlibUtil.java:27)
      at org.jboss.netty.handler.codec.compression.ZlibDecoder.decode(ZlibDecoder.java:153)
      at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:67)
      at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
      at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
      at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
      at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
      at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
      at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:744)

      After some digging I found if I changed 'deflate' for 'gzip' it works.

      agent.sinks.snk1.compression-type = gzip

      agent.sources.src1.compression-type = gzip

      1. patch-2370.diff
        3 kB
        Cesar Delgado

        Activity

        Cesar Delgado created issue -
        Hide
        Cesar Delgado added a comment -

        Changed the docs so others won't get hit by this.

        Show
        Cesar Delgado added a comment - Changed the docs so others won't get hit by this.
        Cesar Delgado made changes -
        Field Original Value New Value
        Attachment patch-2370.diff [ 12640854 ]
        Hide
        Steve Zesch added a comment -

        I've had the AvroSource and a custom Avro Sink using deflate for quite some time now. I do remember seeing a similar errors as yours when we had a mismatch compression setting between source and sink, but your configs look ok.

        Out of curiosity, have you tried using deflate and then specifying a specific compression level? We're currently using a compression level of 9. I doubt it would fix your problem, but it's worth a shot.

        Show
        Steve Zesch added a comment - I've had the AvroSource and a custom Avro Sink using deflate for quite some time now. I do remember seeing a similar errors as yours when we had a mismatch compression setting between source and sink, but your configs look ok. Out of curiosity, have you tried using deflate and then specifying a specific compression level? We're currently using a compression level of 9. I doubt it would fix your problem, but it's worth a shot.
        Hide
        Cesar Delgado added a comment -

        Hey Steve, thanks for the input. The config file does have the compression set. I tried setting it to 1, 6, 9 to cover my bases as part of the testing. Things only started working when I switch it over to gz. Strange.

        Show
        Cesar Delgado added a comment - Hey Steve, thanks for the input. The config file does have the compression set. I tried setting it to 1, 6, 9 to cover my bases as part of the testing. Things only started working when I switch it over to gz. Strange.
        Hide
        Roshan Naik added a comment - - edited

        Cesar Delgado something is not right here.

        I do see the below line of code in AvrosSource.java ..

        boolean enableCompression = compressionType.equalsIgnoreCase("deflate");
        

        But no mention of 'gzip'

        Similarly in the avro sink side in NettyAvroRpcClient.java

        if (enableCompressionStr != null && enableCompressionStr.equalsIgnoreCase("deflate"))
        
        Show
        Roshan Naik added a comment - - edited Cesar Delgado something is not right here. I do see the below line of code in AvrosSource.java .. boolean enableCompression = compressionType.equalsIgnoreCase( "deflate" ); But no mention of 'gzip' Similarly in the avro sink side in NettyAvroRpcClient.java if (enableCompressionStr != null && enableCompressionStr.equalsIgnoreCase( "deflate" ))
        Hide
        Cesar Delgado added a comment -

        Correct, I found the same line. Even the Avro docs mention deflate but not gzip (http://avro.apache.org/docs/1.7.3/spec.html#Required+Codecs) but I'm still getting that error when I change it back.

        One thing that just came to mind is, maybe the compression getting skipped because 'gzip' is not being found as a valid parameter. I'll try to setup a system where I can test the data going though the network with and without the parameter in the next couple of days.

        Show
        Cesar Delgado added a comment - Correct, I found the same line. Even the Avro docs mention deflate but not gzip ( http://avro.apache.org/docs/1.7.3/spec.html#Required+Codecs ) but I'm still getting that error when I change it back. One thing that just came to mind is, maybe the compression getting skipped because 'gzip' is not being found as a valid parameter. I'll try to setup a system where I can test the data going though the network with and without the parameter in the next couple of days.
        Hide
        Roshan Naik added a comment -

        yes with gzip specified the compression would get disabled.

        Show
        Roshan Naik added a comment - yes with gzip specified the compression would get disabled.
        Hide
        Cesar Delgado added a comment -

        I moved all configurations files to a safe place, un-installed all Flume packaged on the machine. Then I re-installed the packages and move the configurations back and now deflate is working for me. Not sure what happened here but it's working now so I'll close this out.

        Show
        Cesar Delgado added a comment - I moved all configurations files to a safe place, un-installed all Flume packaged on the machine. Then I re-installed the packages and move the configurations back and now deflate is working for me. Not sure what happened here but it's working now so I'll close this out.
        Cesar Delgado made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Cannot Reproduce [ 5 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Cesar Delgado
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development