Flume
  1. Flume
  2. FLUME-946

Allow multiplexing channel selector to specify optional channels.

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: v1.0.0, v1.1.0
    • Fix Version/s: v1.3.0
    • Component/s: None
    • Labels:
      None

      Description

      Right now the multiplexing channel selector treats all channels as required channels. There should be a simple provision where some channel can be marked optional so that a failure in publishing the event to that particular channel does not cause the entire request to fail. For example, the following configuration:

      agent.sources.foo.selector.mapping.foo = ch1 ch2
      agent.sources.foo.selector.mapping.bar = ch2 ch3
      agent.sources.foo.selector.optional.channels = ch2

      Would imply that for mapping value "foo", ch1 is a required channel whereas ch2 is an optional channel. Similarly, for mapping "bar', ch3 is a required channel but ch2 is optional.

      The implementation in the patch is:

      I did not implement it exactly as defined in the Jira, instead:
      agent.sources.foo.selector.mapping.zebra = ch2
      agent.sources.foo.selector.optional.zebra = ch1

      Therefore the behavior of "mapping" keyword does not change, instead the optional keyword simply adds more channels as optional channels. If "mapping" keyword does not exist for the header or has no values, then the event is written to default channel and optional channels. If optional is empty then even is simply written to channels specificied by "mapping"(if empty, then to default channels). Basically mapping and optional means: write to channels specified by "mapping" and if available, to the ones specified by "optional." If mapping is empty write to default, and optionally to optional channels.

      1. FLUME-946-3.patch
        10 kB
        Hari Shreedharan
      2. FLUME-946-2.patch
        5 kB
        Hari Shreedharan

        Activity

        Arvind Prabhakar created issue -
        Arvind Prabhakar made changes -
        Field Original Value New Value
        Affects Version/s v1.0.0 [ 12318896 ]
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4034/
        -----------------------------------------------------------

        Review request for Flume.

        Summary
        -------

        Support for optional channels in multiplexing channel selector. If no required channels configured, but there are optional channels then it means, write to default channels and optionally write to optional channels if they are available.

        This addresses bug FLUME-946.
        https://issues.apache.org/jira/browse/FLUME-946

        Diffs


        flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java c63d0a1
        flume-ng-core/src/test/java/org/apache/flume/channel/TestMultiplexingChannelSelector.java 2626b20

        Diff: https://reviews.apache.org/r/4034/diff

        Testing
        -------

        Updated unit tests to support these cases

        Thanks,

        Hari

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4034/ ----------------------------------------------------------- Review request for Flume. Summary ------- Support for optional channels in multiplexing channel selector. If no required channels configured, but there are optional channels then it means, write to default channels and optionally write to optional channels if they are available. This addresses bug FLUME-946 . https://issues.apache.org/jira/browse/FLUME-946 Diffs flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java c63d0a1 flume-ng-core/src/test/java/org/apache/flume/channel/TestMultiplexingChannelSelector.java 2626b20 Diff: https://reviews.apache.org/r/4034/diff Testing ------- Updated unit tests to support these cases Thanks, Hari
        Hari Shreedharan made changes -
        Attachment FLUME-946-2.patch [ 12515953 ]
        Hari Shreedharan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hari Shreedharan made changes -
        Fix Version/s v1.1.0 [ 12319284 ]
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4034/
        -----------------------------------------------------------

        (Updated 2012-02-24 19:00:14.512832)

        Review request for Flume.

        Changes
        -------

        Updated unit test to get optional channel from an event which does not have required channels.

        Summary
        -------

        Support for optional channels in multiplexing channel selector. If no required channels configured, but there are optional channels then it means, write to default channels and optionally write to optional channels if they are available.

        This addresses bug FLUME-946.
        https://issues.apache.org/jira/browse/FLUME-946

        Diffs (updated)


        flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java c63d0a1
        flume-ng-core/src/test/java/org/apache/flume/channel/TestMultiplexingChannelSelector.java 2626b20

        Diff: https://reviews.apache.org/r/4034/diff

        Testing
        -------

        Updated unit tests to support these cases

        Thanks,

        Hari

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4034/ ----------------------------------------------------------- (Updated 2012-02-24 19:00:14.512832) Review request for Flume. Changes ------- Updated unit test to get optional channel from an event which does not have required channels. Summary ------- Support for optional channels in multiplexing channel selector. If no required channels configured, but there are optional channels then it means, write to default channels and optionally write to optional channels if they are available. This addresses bug FLUME-946 . https://issues.apache.org/jira/browse/FLUME-946 Diffs (updated) flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java c63d0a1 flume-ng-core/src/test/java/org/apache/flume/channel/TestMultiplexingChannelSelector.java 2626b20 Diff: https://reviews.apache.org/r/4034/diff Testing ------- Updated unit tests to support these cases Thanks, Hari
        Hide
        Hari Shreedharan added a comment -

        I did not implement it exactly as defined in the Jira, instead:
        agent.sources.foo.selector.mapping.zebra = ch2
        agent.sources.foo.selector.optional.zebra = ch1

        Therefore the behavior of "mapping" keyword does not change, instead the optional keyword simply adds more channels as optional channels. If "mapping" keyword does not exist for the header or has no values, then the event is written to default channel and optional channels. If optional is empty then even is simply written to channels specificied by "mapping"(if empty, then to default channels). Basically mapping and optional means: write to channels specified by "mapping" and if available, to the ones specified by "optional." If mapping is empty write to default, and optionally to optional channels.

        Show
        Hari Shreedharan added a comment - I did not implement it exactly as defined in the Jira, instead: agent.sources.foo.selector.mapping.zebra = ch2 agent.sources.foo.selector.optional.zebra = ch1 Therefore the behavior of "mapping" keyword does not change, instead the optional keyword simply adds more channels as optional channels. If "mapping" keyword does not exist for the header or has no values, then the event is written to default channel and optional channels. If optional is empty then even is simply written to channels specificied by "mapping"(if empty, then to default channels). Basically mapping and optional means: write to channels specified by "mapping" and if available, to the ones specified by "optional." If mapping is empty write to default, and optionally to optional channels.
        Hari Shreedharan made changes -
        Description Right now the multiplexing channel selector treats all channels as required channels. There should be a simple provision where some channel can be marked optional so that a failure in publishing the event to that particular channel does not cause the entire request to fail. For example, the following configuration:

        agent.sources.foo.selector.mapping.foo = ch1 ch2
        agent.sources.foo.selector.mapping.bar = ch2 ch3
        agent.sources.foo.selector.optional.channels = ch2

        Would imply that for mapping value "foo", ch1 is a required channel whereas ch2 is an optional channel. Similarly, for mapping "bar', ch3 is a required channel but ch2 is optional.
        Right now the multiplexing channel selector treats all channels as required channels. There should be a simple provision where some channel can be marked optional so that a failure in publishing the event to that particular channel does not cause the entire request to fail. For example, the following configuration:

        agent.sources.foo.selector.mapping.foo = ch1 ch2
        agent.sources.foo.selector.mapping.bar = ch2 ch3
        agent.sources.foo.selector.optional.channels = ch2

        Would imply that for mapping value "foo", ch1 is a required channel whereas ch2 is an optional channel. Similarly, for mapping "bar', ch3 is a required channel but ch2 is optional.

        The implementation in the patch is:

        I did not implement it exactly as defined in the Jira, instead:
        agent.sources.foo.selector.mapping.zebra = ch2
        agent.sources.foo.selector.optional.zebra = ch1

        Therefore the behavior of "mapping" keyword does not change, instead the optional keyword simply adds more channels as optional channels. If "mapping" keyword does not exist for the header or has no values, then the event is written to default channel and optional channels. If optional is empty then even is simply written to channels specificied by "mapping"(if empty, then to default channels). Basically mapping and optional means: write to channels specified by "mapping" and if available, to the ones specified by "optional." If mapping is empty write to default, and optionally to optional channels.
        Hari Shreedharan made changes -
        Assignee Hari Shreedharan [ hshreedharan ]
        Arvind Prabhakar made changes -
        Fix Version/s v1.2.0 [ 12320243 ]
        Fix Version/s v1.1.0 [ 12319284 ]
        Hari Shreedharan made changes -
        Original Estimate 0h [ 0 ]
        Remaining Estimate 0h [ 0 ]
        Affects Version/s v1.1.0 [ 12319284 ]
        Priority Major [ 3 ] Minor [ 4 ]
        Mike Percy made changes -
        Fix Version/s v1.3.0 [ 12322140 ]
        Fix Version/s v1.2.0 [ 12320243 ]
        Brock Noland made changes -
        Fix Version/s v1.4.0 [ 12323372 ]
        Fix Version/s v1.3.0 [ 12322140 ]
        Hide
        Hari Shreedharan added a comment -

        Committers:

        I submitted this quite a while back, I am not entirely sure if it will apply cleanly. But if it does, could you please review and commit? I see a good use case for this.

        Show
        Hari Shreedharan added a comment - Committers: I submitted this quite a while back, I am not entirely sure if it will apply cleanly. But if it does, could you please review and commit? I see a good use case for this.
        Hari Shreedharan made changes -
        Attachment FLUME-946-3.patch [ 12551230 ]
        Brock Noland made changes -
        Fix Version/s v1.3.0 [ 12322140 ]
        Fix Version/s v1.4.0 [ 12323372 ]
        Hide
        Brock Noland added a comment -

        Committed to trunk, 1.4, and 1.3. Thank you for your patch Hari!

        Show
        Brock Noland added a comment - Committed to trunk, 1.4, and 1.3. Thank you for your patch Hari!
        Brock Noland made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in flume-trunk #320 (See https://builds.apache.org/job/flume-trunk/320/)
        FLUME-946: Allow multiplexing channel selector to specify optional channels (Revision babdb69a3eebdd4fca7f5edd30e6de1d0310d5bd)

        Result = SUCCESS
        brock : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=babdb69a3eebdd4fca7f5edd30e6de1d0310d5bd
        Files :

        • flume-ng-doc/sphinx/FlumeUserGuide.rst
        • flume-ng-core/src/test/java/org/apache/flume/channel/TestMultiplexingChannelSelector.java
        • flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java
        Show
        Hudson added a comment - Integrated in flume-trunk #320 (See https://builds.apache.org/job/flume-trunk/320/ ) FLUME-946 : Allow multiplexing channel selector to specify optional channels (Revision babdb69a3eebdd4fca7f5edd30e6de1d0310d5bd) Result = SUCCESS brock : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=babdb69a3eebdd4fca7f5edd30e6de1d0310d5bd Files : flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-core/src/test/java/org/apache/flume/channel/TestMultiplexingChannelSelector.java flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java

          People

          • Assignee:
            Hari Shreedharan
            Reporter:
            Arvind Prabhakar
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development