Flume
  1. Flume
  2. FLUME-1488

Load Balancing RPC client should support exponential backoff of failed nodes

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.2.0
    • Fix Version/s: v1.3.0
    • Component/s: None
    • Labels:
      None

      Description

      The Load Balancing RPC client should be able to support exponential backoff of failed nodes so that we don't get repeated failures on known bad endpoints.

      1. FLUME-1488-0.patch
        39 kB
        Hari Shreedharan
      2. FLUME-1488-1.patch
        40 kB
        Hari Shreedharan

        Issue Links

          Activity

          Hide
          Hari Shreedharan added a comment -

          I'm planning to implement this. If no one has objections, I am going to pick this up.

          Show
          Hari Shreedharan added a comment - I'm planning to implement this. If no one has objections, I am going to pick this up.
          Hide
          Hari Shreedharan added a comment -

          I have a rough implementation that uses the same code for both Sink selector and rpc client, which as of now uses cut-paste code. Unfortunately, I need to keep the current interfaces as a wrapper to the generic class I implemented because these interfaces were in release 1.2.0 and removing/modifying them or the AbstractSinkSelector class can break custom components. So I am working around this for now.

          Show
          Hari Shreedharan added a comment - I have a rough implementation that uses the same code for both Sink selector and rpc client, which as of now uses cut-paste code. Unfortunately, I need to keep the current interfaces as a wrapper to the generic class I implemented because these interfaces were in release 1.2.0 and removing/modifying them or the AbstractSinkSelector class can break custom components. So I am working around this for now.
          Hide
          Mike Percy added a comment -

          Patch committed. Thanks Hari!

          Rev: 960d03d8a5d3b56db15dd390435a3b359f3ac4fb

          Show
          Mike Percy added a comment - Patch committed. Thanks Hari! Rev: 960d03d8a5d3b56db15dd390435a3b359f3ac4fb
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #302 (See https://builds.apache.org/job/flume-trunk/302/)
          FLUME-1488. Load Balancing RPC client should support exponential backoff of failed nodes. (Revision 960d03d8a5d3b56db15dd390435a3b359f3ac4fb)

          Result = SUCCESS
          mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=960d03d8a5d3b56db15dd390435a3b359f3ac4fb
          Files :

          • flume-ng-sdk/src/main/java/org/apache/flume/util/OrderSelector.java
          • flume-ng-sdk/src/main/java/org/apache/flume/api/LoadBalancingRpcClient.java
          • flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSinkSelector.java
          • flume-ng-sdk/pom.xml
          • flume-ng-core/src/test/java/org/apache/flume/sink/TestLoadBalancingSinkProcessor.java
          • flume-ng-sdk/src/test/java/org/apache/flume/api/TestLoadBalancingRpcClient.java
          • flume-ng-sdk/src/main/java/org/apache/flume/util/RandomOrderSelector.java
          • flume-ng-sdk/src/main/java/org/apache/flume/util/RoundRobinOrderSelector.java
          • flume-ng-core/src/main/java/org/apache/flume/sink/LoadBalancingSinkProcessor.java
          • flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClientConfigurationConstants.java
          Show
          Hudson added a comment - Integrated in flume-trunk #302 (See https://builds.apache.org/job/flume-trunk/302/ ) FLUME-1488 . Load Balancing RPC client should support exponential backoff of failed nodes. (Revision 960d03d8a5d3b56db15dd390435a3b359f3ac4fb) Result = SUCCESS mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=960d03d8a5d3b56db15dd390435a3b359f3ac4fb Files : flume-ng-sdk/src/main/java/org/apache/flume/util/OrderSelector.java flume-ng-sdk/src/main/java/org/apache/flume/api/LoadBalancingRpcClient.java flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSinkSelector.java flume-ng-sdk/pom.xml flume-ng-core/src/test/java/org/apache/flume/sink/TestLoadBalancingSinkProcessor.java flume-ng-sdk/src/test/java/org/apache/flume/api/TestLoadBalancingRpcClient.java flume-ng-sdk/src/main/java/org/apache/flume/util/RandomOrderSelector.java flume-ng-sdk/src/main/java/org/apache/flume/util/RoundRobinOrderSelector.java flume-ng-core/src/main/java/org/apache/flume/sink/LoadBalancingSinkProcessor.java flume-ng-sdk/src/main/java/org/apache/flume/api/RpcClientConfigurationConstants.java
          Hide
          Eugene Koontz added a comment -

          FLUME-2229 fixes a bug in FLUME-1488.

          Show
          Eugene Koontz added a comment - FLUME-2229 fixes a bug in FLUME-1488 .

            People

            • Assignee:
              Hari Shreedharan
              Reporter:
              Kathleen Ting
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development