Flume
  1. Flume
  2. FLUME-1915

Enhance NettyAvroRpcClient and the use of NettyServer to optionally use compression

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.4.0
    • Component/s: None
    • Labels:
      None

      Description

      This JIRA will update AvroSource and NettyAvroRpcClient to optionally use ZlibEncoder and ZlibDecoder to compress communications between themselves.

      1. FLUME-1915-1.patch
        40 kB
        Ted Malaska
      2. FLUME-1915.patch
        26 kB
        Ted Malaska

        Issue Links

          Activity

          Ted Malaska created issue -
          Hari Shreedharan made changes -
          Field Original Value New Value
          Link This issue is duplicated by FLUME-1913 [ FLUME-1913 ]
          Hide
          Ted Malaska added a comment -

          Added compression options for both client and server.

          Note compress has be turn on both client and server for it to work.

          Client can have compression with out compression if compression level is set to 0

          Show
          Ted Malaska added a comment - Added compression options for both client and server. Note compress has be turn on both client and server for it to work. Client can have compression with out compression if compression level is set to 0
          Ted Malaska made changes -
          Attachment FLUME-1915.patch [ 12569220 ]
          Ted Malaska made changes -
          Remote Link This issue links to "Review Board (Web Link)" [ 12021 ]
          Ted Malaska made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Ted Malaska added a comment -

          Updated based on review

          Show
          Ted Malaska added a comment - Updated based on review
          Ted Malaska made changes -
          Attachment FLUME-1915-1.patch [ 12571136 ]
          Hide
          Mike Percy added a comment -

          +1

          Show
          Mike Percy added a comment - +1
          Hide
          Mike Percy added a comment -

          Pushed to trunk and flume-1.4 branches. Thanks for the patch, Ted!

          Show
          Mike Percy added a comment - Pushed to trunk and flume-1.4 branches. Thanks for the patch, Ted!
          Mike Percy made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s v1.4.0 [ 12323372 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #369 (See https://builds.apache.org/job/flume-trunk/369/)
          FLUME-1915. Deflate compression support for the AvroSource, AvroSink, and Avro RPC Client. (Revision e72e559baa589787bcba233e29882e28e0ff43ef)

          Result = SUCCESS
          mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=e72e559baa589787bcba233e29882e28e0ff43ef
          Files :

          • flume-ng-core/src/main/java/org/apache/flume/sink/AbstractRpcSink.java
          • flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java
          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          • flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java
          • flume-ng-sdk/src/test/java/org/apache/flume/api/RpcTestUtils.java
          • flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClientConfigurationConstants.java
          • flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java
          • flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java
          • flume-ng-sdk/src/main/java/org/apache/flume/api/NettyAvroRpcClient.java
          Show
          Hudson added a comment - Integrated in flume-trunk #369 (See https://builds.apache.org/job/flume-trunk/369/ ) FLUME-1915 . Deflate compression support for the AvroSource, AvroSink, and Avro RPC Client. (Revision e72e559baa589787bcba233e29882e28e0ff43ef) Result = SUCCESS mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=e72e559baa589787bcba233e29882e28e0ff43ef Files : flume-ng-core/src/main/java/org/apache/flume/sink/AbstractRpcSink.java flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java flume-ng-sdk/src/test/java/org/apache/flume/api/RpcTestUtils.java flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClientConfigurationConstants.java flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java flume-ng-sdk/src/main/java/org/apache/flume/api/NettyAvroRpcClient.java
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          2d 2h 24m 1 Ted Malaska 15/Feb/13 00:44
          Patch Available Patch Available Resolved Resolved
          21d 12m 1 Mike Percy 08/Mar/13 00:56

            People

            • Assignee:
              Ted Malaska
              Reporter:
              Ted Malaska
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development