Flume
  1. Flume
  2. FLUME-1541

Implement a SinkSelector for LoadBalancingSinkProcessor that includes failover mechanics

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.3.0
    • Fix Version/s: v1.3.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Implement a new sink selector with the following behaviors:

      • Round robin usage of sinks
      • Temporarily removing sinks from the round robin pool when they fail
      • Increasing timeout mechanics on sequential failures
      • Spreading load equally over surviving sinks(current Round Robin sends all the failed sinks events to the next sink in line)
      1. FLUME-1541-3.patch
        20 kB
        Juhani Connolly

        Issue Links

          Activity

          Juhani Connolly created issue -
          Juhani Connolly made changes -
          Field Original Value New Value
          Fix Version/s v1.3.0 [ 12322140 ]
          Affects Version/s v1.3.0 [ 12322140 ]
          Component/s Sinks+Sources [ 12315320 ]
          Juhani Connolly made changes -
          Remote Link This issue links to "Review request (Web Link)" [ 11106 ]
          Hide
          Hari Shreedharan added a comment -

          Juhani,

          This is excellent. Do you think you could add this backoff mechanism to the SDK load balancing Rpc client also. It might just be copy paste code for now, but it would be good to refactor the load balancing code in the RpcClient and Sinks to use the same libraries.

          Show
          Hari Shreedharan added a comment - Juhani, This is excellent. Do you think you could add this backoff mechanism to the SDK load balancing Rpc client also. It might just be copy paste code for now, but it would be good to refactor the load balancing code in the RpcClient and Sinks to use the same libraries.
          Hide
          Juhani Connolly added a comment -

          I can have a look into it when I can get a bit of time. I haven't really touched the sdk at all, but I'm sure it should be ok

          Show
          Juhani Connolly added a comment - I can have a look into it when I can get a bit of time. I haven't really touched the sdk at all, but I'm sure it should be ok
          Juhani Connolly made changes -
          Attachment FLUME-1541-3.patch [ 12544016 ]
          Hide
          Juhani Connolly added a comment -

          Putting this up in advance as RB looks like it should be ok after latest changes

          Show
          Juhani Connolly added a comment - Putting this up in advance as RB looks like it should be ok after latest changes
          Juhani Connolly made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Juhani Connolly [ juhanic ]
          Hide
          Mike Percy added a comment -

          Patch committed. Thanks Juhani!

          Rev: d36861bcea4e3afeb08bfddd46ce0f7533186f00

          Show
          Mike Percy added a comment - Patch committed. Thanks Juhani! Rev: d36861bcea4e3afeb08bfddd46ce0f7533186f00
          Mike Percy made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in flume-1.3.0 #7 (See https://builds.apache.org/job/flume-1.3.0/7/)
          FLUME-1541. Implement failover for LoadBalancingSinkProcessor. (Revision 752f70f4fd053fa53650c0d7673a99c7610464f9)

          Result = FAILURE
          mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=752f70f4fd053fa53650c0d7673a99c7610464f9
          Files :

          • flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSinkSelector.java
          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          • flume-ng-core/src/test/java/org/apache/flume/sink/TestLoadBalancingSinkProcessor.java
          • flume-ng-core/src/main/java/org/apache/flume/sink/LoadBalancingSinkProcessor.java
          Show
          Hudson added a comment - Integrated in flume-1.3.0 #7 (See https://builds.apache.org/job/flume-1.3.0/7/ ) FLUME-1541 . Implement failover for LoadBalancingSinkProcessor. (Revision 752f70f4fd053fa53650c0d7673a99c7610464f9) Result = FAILURE mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=752f70f4fd053fa53650c0d7673a99c7610464f9 Files : flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSinkSelector.java flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-core/src/test/java/org/apache/flume/sink/TestLoadBalancingSinkProcessor.java flume-ng-core/src/main/java/org/apache/flume/sink/LoadBalancingSinkProcessor.java
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #293 (See https://builds.apache.org/job/flume-trunk/293/)
          FLUME-1541. Implement failover for LoadBalancingSinkProcessor. (Revision d36861bcea4e3afeb08bfddd46ce0f7533186f00)

          Result = UNSTABLE
          mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=d36861bcea4e3afeb08bfddd46ce0f7533186f00
          Files :

          • flume-ng-core/src/test/java/org/apache/flume/sink/TestLoadBalancingSinkProcessor.java
          • flume-ng-core/src/main/java/org/apache/flume/sink/LoadBalancingSinkProcessor.java
          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          • flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSinkSelector.java
          Show
          Hudson added a comment - Integrated in flume-trunk #293 (See https://builds.apache.org/job/flume-trunk/293/ ) FLUME-1541 . Implement failover for LoadBalancingSinkProcessor. (Revision d36861bcea4e3afeb08bfddd46ce0f7533186f00) Result = UNSTABLE mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=d36861bcea4e3afeb08bfddd46ce0f7533186f00 Files : flume-ng-core/src/test/java/org/apache/flume/sink/TestLoadBalancingSinkProcessor.java flume-ng-core/src/main/java/org/apache/flume/sink/LoadBalancingSinkProcessor.java flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSinkSelector.java

            People

            • Assignee:
              Juhani Connolly
              Reporter:
              Juhani Connolly
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development