Details

    • Type: New Feature New Feature
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v0.9.5
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Release Note:
      Implemented RoundRobinSink

      Description

      Create a sink like a fanout sink, but instead of sending each event to all of the fanout-sinks, it sends each event to one of the fanout events, preferably in a manner that distributes the load across the sinks evenly.

      1. flume-527.patch
        11 kB
        Frank Grimes
      2. flume-527b.patch
        11 kB
        Frank Grimes
      3. flume-527c.patch
        11 kB
        Frank Grimes

        Activity

        Hide
        Frank Grimes added a comment -

        New patch which changes the round robin sink start and end tokens to '<|' and '|>'

        Show
        Frank Grimes added a comment - New patch which changes the round robin sink start and end tokens to '<|' and '|>'
        Hide
        Jonathan Hsieh added a comment -

        Frank, what do you think of something like '<|' '|>' , or maybe '</' '/>' instead of '
        ' and '/'?

        The '<' and '>' tokens wrapping a failover already have an "OR" connotation, and adding '|' or '/' feels like parallel or splitting which the round round also feels like...

        Show
        Jonathan Hsieh added a comment - Frank, what do you think of something like '<|' '|>' , or maybe '</' '/>' instead of ' ' and '/'? The '<' and '>' tokens wrapping a failover already have an "OR" connotation, and adding '|' or '/' feels like parallel or splitting which the round round also feels like...
        Hide
        Frank Grimes added a comment -

        New patch which uses Apache license header instead of Cloudera, cleans up comments and fixes ambiguous start/end tokens.

        Show
        Frank Grimes added a comment - New patch which uses Apache license header instead of Cloudera, cleans up comments and fixes ambiguous start/end tokens.
        Hide
        Jonathan Hsieh added a comment -

        Frank, great patch! I think there a few think about from looking at the code before I try it:

        How does this parse? – I think we many needs a different start and end tokens.

        1. a b # c # d e #

        Update copyright RoundRobinSink.java to the standard apache one instead of cloudera. Here's an example:
        https://github.com/apache/flume/blob/03aaccfea821e685ff1a23a8330c2f47449d656e/flume-core/src/main/java/com/cloudera/util/HttpServerTestUtils.java

        Some comment nits – get rid of the TODO (jon). (its legacy stuff).

        Would be great if you could document what happens if something fails to open or blocks when attempting to send to something that fails to open? Another thougher question is what should happen when there is a failure (runtime, io, or interrupted),

        Show
        Jonathan Hsieh added a comment - Frank, great patch! I think there a few think about from looking at the code before I try it: How does this parse? – I think we many needs a different start and end tokens. a b # c # d e # Update copyright RoundRobinSink.java to the standard apache one instead of cloudera. Here's an example: https://github.com/apache/flume/blob/03aaccfea821e685ff1a23a8330c2f47449d656e/flume-core/src/main/java/com/cloudera/util/HttpServerTestUtils.java Some comment nits – get rid of the TODO (jon). (its legacy stuff). Would be great if you could document what happens if something fails to open or blocks when attempting to send to something that fails to open? Another thougher question is what should happen when there is a failure (runtime, io, or interrupted),
        Hide
        Frank Grimes added a comment -

        We had a need for this feature as well to distribute decorator CPU load among multiple flows/threads within one collector node.
        The attached patch is based off of trunk@03aaccfea821e685ff1a23a8330c2f47449d656e in git://git.apache.org/flume.git .

        Show
        Frank Grimes added a comment - We had a need for this feature as well to distribute decorator CPU load among multiple flows/threads within one collector node. The attached patch is based off of trunk@03aaccfea821e685ff1a23a8330c2f47449d656e in git://git.apache.org/flume.git .

          People

          • Assignee:
            Frank Grimes
            Reporter:
            Disabled imported user
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development