Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2202

AsyncHBaseSink should coalesce increments to reduce RPC roundtrips

        Details

        • Type: Improvement
        • Status: Resolved
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: 1.5.0
        • Component/s: None
        • Labels:
          None
        1. Text File
          FLUME-2202.patch
          16 kB
          Hari Shreedharan
        2. Text File
          FLUME-2202-2.patch
          20 kB
          Hari Shreedharan
        3. Text File
          FLUME-2202-3.patch
          22 kB
          Hari Shreedharan

          Issue Links

          links to

          Web Link Review

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            hshreedharan Hari Shreedharan added a comment -

            Added a new config to enable coalescing increments per batch. Even though asynchbase has an API that can coalesce, that does not allow us to check how many increments are in the buffer, which we need to calculate how many callbacks to expect - which is used to calculate timeouts.

            Show
            hshreedharan Hari Shreedharan added a comment - Added a new config to enable coalescing increments per batch. Even though asynchbase has an API that can coalesce, that does not allow us to check how many increments are in the buffer, which we need to calculate how many callbacks to expect - which is used to calculate timeouts.
            Hide
            Permalink
            mpercy Mike Percy added a comment -

            +1

            Show
            mpercy Mike Percy added a comment - +1
            Hide
            Permalink
            mpercy Mike Percy added a comment -

            Pushed to trunk and flume-1.5 branches. Thanks for the patch Hari!

            Show
            mpercy Mike Percy added a comment - Pushed to trunk and flume-1.5 branches. Thanks for the patch Hari!
            Hide
            Permalink
            hudson Hudson added a comment -

            FAILURE: Integrated in flume-trunk #507 (See https://builds.apache.org/job/flume-trunk/507/)
            FLUME-2202. AsyncHBaseSink should coalesce increments to reduce RPC roundtrips (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=c4e2129fd12f97303a1b8120a2ecf7da456e1b77)

            • flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/IncrementAsyncHBaseSerializer.java
            • flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java
            • flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkConfigurationConstants.java
            • flume-ng-doc/sphinx/FlumeUserGuide.rst
            • flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestAsyncHBaseSink.java
            Show
            hudson Hudson added a comment - FAILURE: Integrated in flume-trunk #507 (See https://builds.apache.org/job/flume-trunk/507/ ) FLUME-2202 . AsyncHBaseSink should coalesce increments to reduce RPC roundtrips (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=c4e2129fd12f97303a1b8120a2ecf7da456e1b77 ) flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/IncrementAsyncHBaseSerializer.java flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSinkConfigurationConstants.java flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestAsyncHBaseSink.java

              People

              • Assignee:
                hshreedharan Hari Shreedharan
                Reporter:
                hshreedharan Hari Shreedharan
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development