Flume
  1. Flume
  2. FLUME-2202

AsyncHBaseSink should coalesce increments to reduce RPC roundtrips

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.5.0
    • Component/s: None
    • Labels:
      None
    1. FLUME-2202.patch
      16 kB
      Hari Shreedharan
    2. FLUME-2202-2.patch
      20 kB
      Hari Shreedharan
    3. FLUME-2202-3.patch
      22 kB
      Hari Shreedharan

      Issue Links

        Activity

        Hide
        Hudson added a comment -

        FAILURE: Integrated in flume-trunk #507 (See https://builds.apache.org/job/flume-trunk/507/)
        FLUME-2202. AsyncHBaseSink should coalesce increments to reduce RPC roundtrips (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=c4e2129fd12f97303a1b8120a2ecf7da456e1b77)

        • flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/IncrementAsyncHBaseSerializer.java
        • flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java
        • flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkConfigurationConstants.java
        • flume-ng-doc/sphinx/FlumeUserGuide.rst
        • flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestAsyncHBaseSink.java
        Show
        Hudson added a comment - FAILURE: Integrated in flume-trunk #507 (See https://builds.apache.org/job/flume-trunk/507/ ) FLUME-2202 . AsyncHBaseSink should coalesce increments to reduce RPC roundtrips (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=c4e2129fd12f97303a1b8120a2ecf7da456e1b77 ) flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/IncrementAsyncHBaseSerializer.java flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkConfigurationConstants.java flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestAsyncHBaseSink.java
        Hide
        Mike Percy added a comment -

        Pushed to trunk and flume-1.5 branches. Thanks for the patch Hari!

        Show
        Mike Percy added a comment - Pushed to trunk and flume-1.5 branches. Thanks for the patch Hari!
        Hide
        Mike Percy added a comment -

        +1

        Show
        Mike Percy added a comment - +1
        Hide
        Hari Shreedharan added a comment -

        Added a new config to enable coalescing increments per batch. Even though asynchbase has an API that can coalesce, that does not allow us to check how many increments are in the buffer, which we need to calculate how many callbacks to expect - which is used to calculate timeouts.

        Show
        Hari Shreedharan added a comment - Added a new config to enable coalescing increments per batch. Even though asynchbase has an API that can coalesce, that does not allow us to check how many increments are in the buffer, which we need to calculate how many callbacks to expect - which is used to calculate timeouts.

          People

          • Assignee:
            Hari Shreedharan
            Reporter:
            Hari Shreedharan
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development