Flume
  1. Flume
  2. FLUME-918

customizable avro source/sink events

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Currently, flume ships with a flume.avdl that has a reference implementation of a AvroSource / AvroSink , to deal with the default flume transports ( Event / Channel - s) .

      We need to customize the avro schema that goes in, for our purposes - but would still like to reuse a majority of code of AvroSource / AvroSink that does the marshalling/ unmarshalling.

        Activity

        Hide
        Karthik K added a comment -

        Proposed Solution:

        Make AvroSource / AvroSink abstract:

        Default Implementation of AvroSink / AvroSource , preserving existing behavior added.

        Custom implementation can inherit from AvroSource / AvroSink , to define their own protocol and schema.

        Show
        Karthik K added a comment - Proposed Solution: Make AvroSource / AvroSink abstract: Default Implementation of AvroSink / AvroSource , preserving existing behavior added. Custom implementation can inherit from AvroSource / AvroSink , to define their own protocol and schema.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3359/
        -----------------------------------------------------------

        Review request for Flume.

        Summary
        -------

        AvroSource / AvroSink made abstract, that are agnostic of the Avro object used for transport or the interchange between Avro objects and the Flume 'Event's and 'Channel's for communication.

        DefaultAvroSource / DefaultAvroSink in place to preserve existing behavior.

        Custom Avro transports can inherit from AvroSource / AvroSink-s , to override the methods.

        This addresses bug FLUME-918.
        https://issues.apache.org/jira/browse/FLUME-918

        Diffs


        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 1226957
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/DefaultAvroSink.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1226957
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/ChannelEventProcessor.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/DefaultAvroSource.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 1226957
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java 1226957

        Diff: https://reviews.apache.org/r/3359/diff

        Testing
        -------

        Thanks,

        Karthik

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3359/ ----------------------------------------------------------- Review request for Flume. Summary ------- AvroSource / AvroSink made abstract, that are agnostic of the Avro object used for transport or the interchange between Avro objects and the Flume 'Event's and 'Channel's for communication. DefaultAvroSource / DefaultAvroSink in place to preserve existing behavior. Custom Avro transports can inherit from AvroSource / AvroSink-s , to override the methods. This addresses bug FLUME-918 . https://issues.apache.org/jira/browse/FLUME-918 Diffs http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 1226957 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/DefaultAvroSink.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1226957 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/ChannelEventProcessor.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/DefaultAvroSource.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 1226957 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java 1226957 Diff: https://reviews.apache.org/r/3359/diff Testing ------- Thanks, Karthik
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3359/
        -----------------------------------------------------------

        (Updated 2012-01-04 00:32:26.125490)

        Review request for Flume.

        Changes
        -------

        in-place AvroSource::toString() to avoid bogus diff in the diff report.

        Summary
        -------

        AvroSource / AvroSink made abstract, that are agnostic of the Avro object used for transport or the interchange between Avro objects and the Flume 'Event's and 'Channel's for communication.

        DefaultAvroSource / DefaultAvroSink in place to preserve existing behavior.

        Custom Avro transports can inherit from AvroSource / AvroSink-s , to override the methods.

        This addresses bug FLUME-918.
        https://issues.apache.org/jira/browse/FLUME-918

        Diffs (updated)


        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 1226957
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1226957
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 1226957
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java 1226957

        Diff: https://reviews.apache.org/r/3359/diff

        Testing
        -------

        Thanks,

        Karthik

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3359/ ----------------------------------------------------------- (Updated 2012-01-04 00:32:26.125490) Review request for Flume. Changes ------- in-place AvroSource::toString() to avoid bogus diff in the diff report. Summary ------- AvroSource / AvroSink made abstract, that are agnostic of the Avro object used for transport or the interchange between Avro objects and the Flume 'Event's and 'Channel's for communication. DefaultAvroSource / DefaultAvroSink in place to preserve existing behavior. Custom Avro transports can inherit from AvroSource / AvroSink-s , to override the methods. This addresses bug FLUME-918 . https://issues.apache.org/jira/browse/FLUME-918 Diffs (updated) http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 1226957 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1226957 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 1226957 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java 1226957 Diff: https://reviews.apache.org/r/3359/diff Testing ------- Thanks, Karthik
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3359/
        -----------------------------------------------------------

        (Updated 2012-01-04 00:35:14.420595)

        Review request for Flume.

        Changes
        -------

        Adding missing 3 new sourcefiles ( DefaultAvroSource/Sink/ChannelEventProcessor)

        Summary
        -------

        AvroSource / AvroSink made abstract, that are agnostic of the Avro object used for transport or the interchange between Avro objects and the Flume 'Event's and 'Channel's for communication.

        DefaultAvroSource / DefaultAvroSink in place to preserve existing behavior.

        Custom Avro transports can inherit from AvroSource / AvroSink-s , to override the methods.

        This addresses bug FLUME-918.
        https://issues.apache.org/jira/browse/FLUME-918

        Diffs (updated)


        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 1226957
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/DefaultAvroSink.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1226957
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/ChannelEventProcessor.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/DefaultAvroSource.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 1226957
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java 1226957

        Diff: https://reviews.apache.org/r/3359/diff

        Testing
        -------

        Thanks,

        Karthik

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3359/ ----------------------------------------------------------- (Updated 2012-01-04 00:35:14.420595) Review request for Flume. Changes ------- Adding missing 3 new sourcefiles ( DefaultAvroSource/Sink/ChannelEventProcessor) Summary ------- AvroSource / AvroSink made abstract, that are agnostic of the Avro object used for transport or the interchange between Avro objects and the Flume 'Event's and 'Channel's for communication. DefaultAvroSource / DefaultAvroSink in place to preserve existing behavior. Custom Avro transports can inherit from AvroSource / AvroSink-s , to override the methods. This addresses bug FLUME-918 . https://issues.apache.org/jira/browse/FLUME-918 Diffs (updated) http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 1226957 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/DefaultAvroSink.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1226957 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/ChannelEventProcessor.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/DefaultAvroSource.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 1226957 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java 1226957 Diff: https://reviews.apache.org/r/3359/diff Testing ------- Thanks, Karthik
        Hide
        E. Sammer added a comment - - edited

        Karthik:

        I'm looking over your patch now. Generally, I agree with the idea of decoupling the notion of sources and sinks from event processors. I like the idea of having a ChannelEventProcessor (or maybe just EventProcessor). I wonder if we should extend this to be Flume NG's version of decorators. Maybe each source and sink can be configured with an optional list of pre-process and post-process EventProcessors as well as single EventProcessor that actually handles the message. Just thinking out loud.

        I do have some style nits with the review (I'll mark it up), but more importantly, a strong urge to avoid extension by inheritance. What do you think about having ChannelEventProcessor be injected into AvroSource / Sink (which would be concrete, but generic) rather than requiring inheritance?

        Show
        E. Sammer added a comment - - edited Karthik: I'm looking over your patch now. Generally, I agree with the idea of decoupling the notion of sources and sinks from event processors. I like the idea of having a ChannelEventProcessor (or maybe just EventProcessor). I wonder if we should extend this to be Flume NG's version of decorators. Maybe each source and sink can be configured with an optional list of pre-process and post-process EventProcessors as well as single EventProcessor that actually handles the message. Just thinking out loud. I do have some style nits with the review (I'll mark it up), but more importantly, a strong urge to avoid extension by inheritance. What do you think about having ChannelEventProcessor be injected into AvroSource / Sink (which would be concrete, but generic) rather than requiring inheritance?
        Hide
        Karthik K added a comment - - edited

        [quote]
        I wonder if we should extend this to be Flume NG's version of decorators.
        Maybe each source and sink can be configured with an optional list of pre-process and post-process EventProcessors as well as single EventProcessor that actually handles the message
        [/quote]

        I agree, that EventProcessor-s are not specific to AvroSource/Sink but pretty generic. (May be , would belong in the o.a.flume.event , package ? ) .

        [quote]
        What do you think about having ChannelEventProcessor be injected into AvroSource / Sink (which would be concrete, but generic) rather than requiring inheritance?
        [/quote]

        I agree.

        With AvroSource, currently it does management of underlying resources + reference implementation of (AvroSource)Protocol as well.

        Probably, either of those needs to be separated ?

        Show
        Karthik K added a comment - - edited [quote] I wonder if we should extend this to be Flume NG's version of decorators. Maybe each source and sink can be configured with an optional list of pre-process and post-process EventProcessors as well as single EventProcessor that actually handles the message [/quote] I agree, that EventProcessor-s are not specific to AvroSource/Sink but pretty generic. (May be , would belong in the o.a.flume.event , package ? ) . [quote] What do you think about having ChannelEventProcessor be injected into AvroSource / Sink (which would be concrete, but generic) rather than requiring inheritance? [/quote] I agree. With AvroSource, currently it does management of underlying resources + reference implementation of (AvroSource)Protocol as well. Probably, either of those needs to be separated ?
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3359/
        -----------------------------------------------------------

        (Updated 2012-01-04 02:45:47.788788)

        Review request for Flume.

        Changes
        -------

        o.a.flume.event . SourceEventHandler and SinkEventHandler added.

        Default implementation of SinkEventHandler - AvroSourceProtocolSinkHandler implemented.

        AvroSource: Separation of management of resources and reference implementation of AvroSourceProtocol .

        New class - AvroSourceProtocolImpl , implements the same.

        Summary
        -------

        AvroSource / AvroSink made abstract, that are agnostic of the Avro object used for transport or the interchange between Avro objects and the Flume 'Event's and 'Channel's for communication.

        DefaultAvroSource / DefaultAvroSink in place to preserve existing behavior.

        Custom Avro transports can inherit from AvroSource / AvroSink-s , to override the methods.

        This addresses bug FLUME-918.
        https://issues.apache.org/jira/browse/FLUME-918

        Diffs (updated)


        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHandler.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SourceEventHandler.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 1226957
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 1226957
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSourceProtocolSinkHandler.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1226957
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSourceProtocolImpl.java PRE-CREATION

        Diff: https://reviews.apache.org/r/3359/diff

        Testing
        -------

        Thanks,

        Karthik

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3359/ ----------------------------------------------------------- (Updated 2012-01-04 02:45:47.788788) Review request for Flume. Changes ------- o.a.flume.event . SourceEventHandler and SinkEventHandler added. Default implementation of SinkEventHandler - AvroSourceProtocolSinkHandler implemented. AvroSource: Separation of management of resources and reference implementation of AvroSourceProtocol . New class - AvroSourceProtocolImpl , implements the same. Summary ------- AvroSource / AvroSink made abstract, that are agnostic of the Avro object used for transport or the interchange between Avro objects and the Flume 'Event's and 'Channel's for communication. DefaultAvroSource / DefaultAvroSink in place to preserve existing behavior. Custom Avro transports can inherit from AvroSource / AvroSink-s , to override the methods. This addresses bug FLUME-918 . https://issues.apache.org/jira/browse/FLUME-918 Diffs (updated) http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHandler.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SourceEventHandler.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 1226957 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 1226957 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSourceProtocolSinkHandler.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1226957 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSourceProtocolImpl.java PRE-CREATION Diff: https://reviews.apache.org/r/3359/diff Testing ------- Thanks, Karthik
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3359/
        -----------------------------------------------------------

        (Updated 2012-01-04 02:45:57.184579)

        Review request for Flume.

        Summary
        -------

        AvroSource / AvroSink made abstract, that are agnostic of the Avro object used for transport or the interchange between Avro objects and the Flume 'Event's and 'Channel's for communication.

        DefaultAvroSource / DefaultAvroSink in place to preserve existing behavior.

        Custom Avro transports can inherit from AvroSource / AvroSink-s , to override the methods.

        This addresses bug FLUME-918.
        https://issues.apache.org/jira/browse/FLUME-918

        Diffs (updated)


        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHandler.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SourceEventHandler.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 1226957
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 1226957
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSourceProtocolSinkHandler.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1226957
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSourceProtocolImpl.java PRE-CREATION

        Diff: https://reviews.apache.org/r/3359/diff

        Testing
        -------

        Thanks,

        Karthik

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3359/ ----------------------------------------------------------- (Updated 2012-01-04 02:45:57.184579) Review request for Flume. Summary ------- AvroSource / AvroSink made abstract, that are agnostic of the Avro object used for transport or the interchange between Avro objects and the Flume 'Event's and 'Channel's for communication. DefaultAvroSource / DefaultAvroSink in place to preserve existing behavior. Custom Avro transports can inherit from AvroSource / AvroSink-s , to override the methods. This addresses bug FLUME-918 . https://issues.apache.org/jira/browse/FLUME-918 Diffs (updated) http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHandler.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SourceEventHandler.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 1226957 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 1226957 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSourceProtocolSinkHandler.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1226957 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSourceProtocolImpl.java PRE-CREATION Diff: https://reviews.apache.org/r/3359/diff Testing ------- Thanks, Karthik
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3359/
        -----------------------------------------------------------

        (Updated 2012-01-06 19:34:06.860377)

        Review request for Flume.

        Changes
        -------

        public interface SourceEventHandler

        { List<Event> marshallFlumeEvents(); }

        public interface SinkEventHandler

        { void unmarshallFlumeEvents(Object protocolObj, List<Event> batchEvents) throws AvroRemoteException; }

        provided for eventHandlers

        Summary (updated)
        -------

        AvroSource / AvroSink concrete classes has injectible eventHandlers (SourceEventHandler and SinkEventHandler), to allow for diffrerent avro protocols to be sent over flume NG. ( not restricted to AvroSourceProtocol , as it stands today).

        Default implementation refers to AvroSourceProtocol still though.

        This addresses bug FLUME-918.
        https://issues.apache.org/jira/browse/FLUME-918

        Diffs (updated)


        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHandler.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SourceEventHandler.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 1226957
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 1226957
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSourceProtocolSinkHandler.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1226957
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSourceProtocolImpl.java PRE-CREATION

        Diff: https://reviews.apache.org/r/3359/diff

        Testing
        -------

        Thanks,

        Karthik

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3359/ ----------------------------------------------------------- (Updated 2012-01-06 19:34:06.860377) Review request for Flume. Changes ------- public interface SourceEventHandler { List<Event> marshallFlumeEvents(); } public interface SinkEventHandler { void unmarshallFlumeEvents(Object protocolObj, List<Event> batchEvents) throws AvroRemoteException; } provided for eventHandlers Summary (updated) ------- AvroSource / AvroSink concrete classes has injectible eventHandlers (SourceEventHandler and SinkEventHandler), to allow for diffrerent avro protocols to be sent over flume NG. ( not restricted to AvroSourceProtocol , as it stands today). Default implementation refers to AvroSourceProtocol still though. This addresses bug FLUME-918 . https://issues.apache.org/jira/browse/FLUME-918 Diffs (updated) http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHandler.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SourceEventHandler.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 1226957 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 1226957 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSourceProtocolSinkHandler.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1226957 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSourceProtocolImpl.java PRE-CREATION Diff: https://reviews.apache.org/r/3359/diff Testing ------- Thanks, Karthik
        Hide
        Karthik K added a comment -

        Eric -
        Please find attached the revised patch herewith. Can you help review the same for the abstractions / signatures . Thanks.

        Show
        Karthik K added a comment - Eric - Please find attached the revised patch herewith. Can you help review the same for the abstractions / signatures . Thanks.
        Hide
        Karthik K added a comment -

        Revised patch attached herewith, that takes out the transaction based event transmission out of AvroSource and makes it generic.

        Show
        Karthik K added a comment - Revised patch attached herewith, that takes out the transaction based event transmission out of AvroSource and makes it generic.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3359/
        -----------------------------------------------------------

        (Updated 2012-01-19 20:37:02.912881)

        Review request for Flume.

        Changes
        -------

        Revised patch attached that takes out the transaction semantics out of avrosource, and makes it generic. New class - TransactionManager introduced with the semantics.

        Summary
        -------

        AvroSource / AvroSink concrete classes has injectible eventHandlers (SourceEventHandler and SinkEventHandler), to allow for diffrerent avro protocols to be sent over flume NG. ( not restricted to AvroSourceProtocol , as it stands today).

        Default implementation refers to AvroSourceProtocol still though.

        This addresses bug FLUME-918.
        https://issues.apache.org/jira/browse/FLUME-918

        Diffs (updated)


        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHandler.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SourceEventHandler.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 1233531
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 1233531
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSourceProtocolSinkHandler.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1233531
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSourceProtocolImpl.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/TransactionManager.java PRE-CREATION

        Diff: https://reviews.apache.org/r/3359/diff

        Testing
        -------

        Thanks,

        Karthik

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3359/ ----------------------------------------------------------- (Updated 2012-01-19 20:37:02.912881) Review request for Flume. Changes ------- Revised patch attached that takes out the transaction semantics out of avrosource, and makes it generic. New class - TransactionManager introduced with the semantics. Summary ------- AvroSource / AvroSink concrete classes has injectible eventHandlers (SourceEventHandler and SinkEventHandler), to allow for diffrerent avro protocols to be sent over flume NG. ( not restricted to AvroSourceProtocol , as it stands today). Default implementation refers to AvroSourceProtocol still though. This addresses bug FLUME-918 . https://issues.apache.org/jira/browse/FLUME-918 Diffs (updated) http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHandler.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SourceEventHandler.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 1233531 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 1233531 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSourceProtocolSinkHandler.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1233531 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSourceProtocolImpl.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/TransactionManager.java PRE-CREATION Diff: https://reviews.apache.org/r/3359/diff Testing ------- Thanks, Karthik
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3359/
        -----------------------------------------------------------

        (Updated 2012-01-19 20:44:49.976106)

        Review request for Flume.

        Changes
        -------

        Documentation added.

        Summary
        -------

        AvroSource / AvroSink concrete classes has injectible eventHandlers (SourceEventHandler and SinkEventHandler), to allow for diffrerent avro protocols to be sent over flume NG. ( not restricted to AvroSourceProtocol , as it stands today).

        Default implementation refers to AvroSourceProtocol still though.

        This addresses bug FLUME-918.
        https://issues.apache.org/jira/browse/FLUME-918

        Diffs (updated)


        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHandler.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SourceEventHandler.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 1233531
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 1233531
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSourceProtocolSinkHandler.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1233531
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSourceProtocolImpl.java PRE-CREATION
        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/TransactionManager.java PRE-CREATION

        Diff: https://reviews.apache.org/r/3359/diff

        Testing
        -------

        Thanks,

        Karthik

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3359/ ----------------------------------------------------------- (Updated 2012-01-19 20:44:49.976106) Review request for Flume. Changes ------- Documentation added. Summary ------- AvroSource / AvroSink concrete classes has injectible eventHandlers (SourceEventHandler and SinkEventHandler), to allow for diffrerent avro protocols to be sent over flume NG. ( not restricted to AvroSourceProtocol , as it stands today). Default implementation refers to AvroSourceProtocol still though. This addresses bug FLUME-918 . https://issues.apache.org/jira/browse/FLUME-918 Diffs (updated) http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHandler.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SourceEventHandler.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 1233531 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 1233531 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSourceProtocolSinkHandler.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1233531 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSourceProtocolImpl.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/TransactionManager.java PRE-CREATION Diff: https://reviews.apache.org/r/3359/diff Testing ------- Thanks, Karthik
        Hide
        Karthik K added a comment -

        Given the nature of the change, would be useful to hear opinions on the same, as we currently have a lot of 'source/sink' code duplicated and patched. Thanks !

        Show
        Karthik K added a comment - Given the nature of the change, would be useful to hear opinions on the same, as we currently have a lot of 'source/sink' code duplicated and patched. Thanks !
        Hide
        Shumin Wu added a comment -

        My team has a use case that needs customised Avro protocol. It would be nice if this patch could get in. Thanks!

        Show
        Shumin Wu added a comment - My team has a use case that needs customised Avro protocol. It would be nice if this patch could get in. Thanks!
        Hide
        Karthik K added a comment -

        Review check, please ?

        Show
        Karthik K added a comment - Review check, please ?
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3359/#review5040
        -----------------------------------------------------------

        Thanks for the patch Karthik. Can you please rebase the patch on the latest source and add some tests? That will help speed up the review as well. Also, some minor feedback below from a quick glance.

        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java
        <https://reviews.apache.org/r/3359/#comment11068>

        No tabs please. Use spaces instead.

        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSourceProtocolSinkHandler.java
        <https://reviews.apache.org/r/3359/#comment11069>

        Please use setter instead of field access.

        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSourceProtocolSinkHandler.java
        <https://reviews.apache.org/r/3359/#comment11070>

        Please use getter for headers.

        • Arvind

        On 2012-01-19 20:44:49, Karthik K wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/3359/

        -----------------------------------------------------------

        (Updated 2012-01-19 20:44:49)

        Review request for Flume.

        Summary

        -------

        AvroSource / AvroSink concrete classes has injectible eventHandlers (SourceEventHandler and SinkEventHandler), to allow for diffrerent avro protocols to be sent over flume NG. ( not restricted to AvroSourceProtocol , as it stands today).

        Default implementation refers to AvroSourceProtocol still though.

        This addresses bug FLUME-918.

        https://issues.apache.org/jira/browse/FLUME-918

        Diffs

        -----

        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHandler.java PRE-CREATION

        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SourceEventHandler.java PRE-CREATION

        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 1233531

        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 1233531

        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSourceProtocolSinkHandler.java PRE-CREATION

        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1233531

        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSourceProtocolImpl.java PRE-CREATION

        http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/TransactionManager.java PRE-CREATION

        Diff: https://reviews.apache.org/r/3359/diff

        Testing

        -------

        Thanks,

        Karthik

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3359/#review5040 ----------------------------------------------------------- Thanks for the patch Karthik. Can you please rebase the patch on the latest source and add some tests? That will help speed up the review as well. Also, some minor feedback below from a quick glance. http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java < https://reviews.apache.org/r/3359/#comment11068 > No tabs please. Use spaces instead. http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSourceProtocolSinkHandler.java < https://reviews.apache.org/r/3359/#comment11069 > Please use setter instead of field access. http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSourceProtocolSinkHandler.java < https://reviews.apache.org/r/3359/#comment11070 > Please use getter for headers. Arvind On 2012-01-19 20:44:49, Karthik K wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3359/ ----------------------------------------------------------- (Updated 2012-01-19 20:44:49) Review request for Flume. Summary ------- AvroSource / AvroSink concrete classes has injectible eventHandlers (SourceEventHandler and SinkEventHandler), to allow for diffrerent avro protocols to be sent over flume NG. ( not restricted to AvroSourceProtocol , as it stands today). Default implementation refers to AvroSourceProtocol still though. This addresses bug FLUME-918 . https://issues.apache.org/jira/browse/FLUME-918 Diffs ----- http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHandler.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/event/SourceEventHandler.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 1233531 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 1233531 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSourceProtocolSinkHandler.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 1233531 http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSourceProtocolImpl.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/TransactionManager.java PRE-CREATION Diff: https://reviews.apache.org/r/3359/diff Testing ------- Thanks, Karthik
        Hide
        Mike Percy added a comment -

        This JIRA has not had an update in over a year. Canceling patch available and clearing fixVersion for now. Karthik, please update the JIRA if you would like to continue to work on this. Thanks!

        Show
        Mike Percy added a comment - This JIRA has not had an update in over a year. Canceling patch available and clearing fixVersion for now. Karthik, please update the JIRA if you would like to continue to work on this. Thanks!

          People

          • Assignee:
            Karthik K
            Reporter:
            Karthik K
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development