Flume
  1. Flume
  2. FLUME-1097

Support modification of event Headers through configuration

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      As discussed on the flume threads we want to support adding/modifying headers at flume agents through configuration.

      Primary use-case as of now -

      Use HDFSEventSink to publish data in paths like /flume-data/YYYY/mm/DD/HH/MM wherein timestamp is of agent running HDFSEventSink.

      What's not there today
      -------------------------

      1. headers are put by clientLIB and can't be added/modified in flight by agents.

        Issue Links

          Activity

          Hide
          Inder SIngh added a comment -

          Adding support for supporting custom headerDecorators in Sinks through sinkConfiguration

          Show
          Inder SIngh added a comment - Adding support for supporting custom headerDecorators in Sinks through sinkConfiguration
          Hide
          Inder SIngh added a comment -

          Folks,

          Please review https://issues.apache.org/jira/secure/attachment/12523368/FLUME-1097-1.patch which attempts at adding CustomHeaderDecorator support's to Sinks through Sink configuration.

          Overview of Changes

          1.EventHeaderDecorator -> generic class having processHeader()
          2.SinkEventHeaderDecorator is a EventHeaderDecorator which overrides timestamp header at this hop.
          3.EventHeaderDecoratorFactory provides EventHeaderDecorator's based on <sink>.headerDecorator config
          4.AbstractSink -> has a eventHeaderDecortaor
          5.HDFSEventSink -> checks for decorator through config and calls processHeader() for each event which modifies/add headers based on event decorator defined through config.

          Sample Configuration for testing
          ----------------------------------
          agent1.sinks.log-sink1.hdfs.headerDecorator = sinkDecorator

          Testing Done
          -------------
          1. Unit tests
          2. manual tests done - agent1.sinks.log-sink1.hdfs.filePrefix = flume-data/%D results in publishing data with timestamp at agent's hop running HDFSEventSink

          Show
          Inder SIngh added a comment - Folks, Please review https://issues.apache.org/jira/secure/attachment/12523368/FLUME-1097-1.patch which attempts at adding CustomHeaderDecorator support's to Sinks through Sink configuration. Overview of Changes 1.EventHeaderDecorator -> generic class having processHeader() 2.SinkEventHeaderDecorator is a EventHeaderDecorator which overrides timestamp header at this hop. 3.EventHeaderDecoratorFactory provides EventHeaderDecorator's based on <sink>.headerDecorator config 4.AbstractSink -> has a eventHeaderDecortaor 5.HDFSEventSink -> checks for decorator through config and calls processHeader() for each event which modifies/add headers based on event decorator defined through config. Sample Configuration for testing ---------------------------------- agent1.sinks.log-sink1.hdfs.headerDecorator = sinkDecorator Testing Done ------------- 1. Unit tests 2. manual tests done - agent1.sinks.log-sink1.hdfs.filePrefix = flume-data/%D results in publishing data with timestamp at agent's hop running HDFSEventSink
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-04-20 20:07:27, Hari Shreedharan wrote:

          > flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java, line 29

          > <https://reviews.apache.org/r/4826/diff/1/?file=103469#file103469line29>

          >

          > This should be named something like TimeStampEventDecorator or something. We might want to add decorators that add other stuff at the sink.

          Thanks Hari. I'll work towards this feedback and come back.

          • Inder

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4826/#review7092
          -----------------------------------------------------------

          On 2012-04-20 10:33:14, Inder Singh wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4826/

          -----------------------------------------------------------

          (Updated 2012-04-20 10:33:14)

          Review request for Flume, Arvind Prabhakar, Mike Percy, and Hari Shreedharan.

          Summary

          -------

          Overview of Changes

          1.EventHeaderDecorator -> generic class having processHeader()

          2.SinkEventHeaderDecorator is a EventHeaderDecorator which overrides timestamp header at this hop.

          3.EventHeaderDecoratorFactory provides EventHeaderDecorator's based on <sink>.headerDecorator config

          4.AbstractSink -> has a eventHeaderDecortaor

          5.HDFSEventSink -> checks for decorator through config and calls processHeader() for each event which modifies/add headers based on event decorator defined through config.

          Sample Configuration for testing

          ----------------------------------

          agent1.sinks.log-sink1.hdfs.headerDecorator = sinkDecorator

          This addresses bug FLUME-1097.

          https://issues.apache.org/jira/browse/FLUME-1097

          Diffs

          -----

          flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecorator.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecoratorFactory.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059

          flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 1b61cad

          Diff: https://reviews.apache.org/r/4826/diff

          Testing

          -------

          Testing Done

          -------------

          1. Unit tests

          2. manual tests done - agent1.sinks.log-sink1.hdfs.filePrefix = flume-data/%D results in publishing data with timestamp at agent's hop running HDFSEventSink

          Thanks,

          Inder

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-04-20 20:07:27, Hari Shreedharan wrote: > flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java, line 29 > < https://reviews.apache.org/r/4826/diff/1/?file=103469#file103469line29 > > > This should be named something like TimeStampEventDecorator or something. We might want to add decorators that add other stuff at the sink. Thanks Hari. I'll work towards this feedback and come back. Inder ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4826/#review7092 ----------------------------------------------------------- On 2012-04-20 10:33:14, Inder Singh wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4826/ ----------------------------------------------------------- (Updated 2012-04-20 10:33:14) Review request for Flume, Arvind Prabhakar, Mike Percy, and Hari Shreedharan. Summary ------- Overview of Changes 1.EventHeaderDecorator -> generic class having processHeader() 2.SinkEventHeaderDecorator is a EventHeaderDecorator which overrides timestamp header at this hop. 3.EventHeaderDecoratorFactory provides EventHeaderDecorator's based on <sink>.headerDecorator config 4.AbstractSink -> has a eventHeaderDecortaor 5.HDFSEventSink -> checks for decorator through config and calls processHeader() for each event which modifies/add headers based on event decorator defined through config. Sample Configuration for testing ---------------------------------- agent1.sinks.log-sink1.hdfs.headerDecorator = sinkDecorator This addresses bug FLUME-1097 . https://issues.apache.org/jira/browse/FLUME-1097 Diffs ----- flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecorator.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecoratorFactory.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 1b61cad Diff: https://reviews.apache.org/r/4826/diff Testing ------- Testing Done ------------- 1. Unit tests 2. manual tests done - agent1.sinks.log-sink1.hdfs.filePrefix = flume-data/%D results in publishing data with timestamp at agent's hop running HDFSEventSink Thanks, Inder
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4826/#review7102
          -----------------------------------------------------------

          Thanks for the patch, Inder! This looks to be very useful. Below are a few suggestions.

          flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecorator.java
          <https://reviews.apache.org/r/4826/#comment15714>

          This abstract class should probably be an interface (it has no implementation).

          flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecoratorFactory.java
          <https://reviews.apache.org/r/4826/#comment15716>

          Please don't use static variables. Just create a new instance every time from this factory.

          flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java
          <https://reviews.apache.org/r/4826/#comment15717>

          Does this class belong in the core at all?

          In my view, this class isn't general enough to be checked into the Flume core. This should be kept as an application-specific plugin which is added to the runtime classpath and referenced in the config. That's sort of the point of adding this EventHeaderDecorator plugin interface, right?

          • Mike

          On 2012-04-20 10:33:14, Inder Singh wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4826/

          -----------------------------------------------------------

          (Updated 2012-04-20 10:33:14)

          Review request for Flume, Arvind Prabhakar, Mike Percy, and Hari Shreedharan.

          Summary

          -------

          Overview of Changes

          1.EventHeaderDecorator -> generic class having processHeader()

          2.SinkEventHeaderDecorator is a EventHeaderDecorator which overrides timestamp header at this hop.

          3.EventHeaderDecoratorFactory provides EventHeaderDecorator's based on <sink>.headerDecorator config

          4.AbstractSink -> has a eventHeaderDecortaor

          5.HDFSEventSink -> checks for decorator through config and calls processHeader() for each event which modifies/add headers based on event decorator defined through config.

          Sample Configuration for testing

          ----------------------------------

          agent1.sinks.log-sink1.hdfs.headerDecorator = sinkDecorator

          This addresses bug FLUME-1097.

          https://issues.apache.org/jira/browse/FLUME-1097

          Diffs

          -----

          flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecorator.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecoratorFactory.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059

          flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 1b61cad

          Diff: https://reviews.apache.org/r/4826/diff

          Testing

          -------

          Testing Done

          -------------

          1. Unit tests

          2. manual tests done - agent1.sinks.log-sink1.hdfs.filePrefix = flume-data/%D results in publishing data with timestamp at agent's hop running HDFSEventSink

          Thanks,

          Inder

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4826/#review7102 ----------------------------------------------------------- Thanks for the patch, Inder! This looks to be very useful. Below are a few suggestions. flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecorator.java < https://reviews.apache.org/r/4826/#comment15714 > This abstract class should probably be an interface (it has no implementation). flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecoratorFactory.java < https://reviews.apache.org/r/4826/#comment15716 > Please don't use static variables. Just create a new instance every time from this factory. flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java < https://reviews.apache.org/r/4826/#comment15717 > Does this class belong in the core at all? In my view, this class isn't general enough to be checked into the Flume core. This should be kept as an application-specific plugin which is added to the runtime classpath and referenced in the config. That's sort of the point of adding this EventHeaderDecorator plugin interface, right? Mike On 2012-04-20 10:33:14, Inder Singh wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4826/ ----------------------------------------------------------- (Updated 2012-04-20 10:33:14) Review request for Flume, Arvind Prabhakar, Mike Percy, and Hari Shreedharan. Summary ------- Overview of Changes 1.EventHeaderDecorator -> generic class having processHeader() 2.SinkEventHeaderDecorator is a EventHeaderDecorator which overrides timestamp header at this hop. 3.EventHeaderDecoratorFactory provides EventHeaderDecorator's based on <sink>.headerDecorator config 4.AbstractSink -> has a eventHeaderDecortaor 5.HDFSEventSink -> checks for decorator through config and calls processHeader() for each event which modifies/add headers based on event decorator defined through config. Sample Configuration for testing ---------------------------------- agent1.sinks.log-sink1.hdfs.headerDecorator = sinkDecorator This addresses bug FLUME-1097 . https://issues.apache.org/jira/browse/FLUME-1097 Diffs ----- flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecorator.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecoratorFactory.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 1b61cad Diff: https://reviews.apache.org/r/4826/diff Testing ------- Testing Done ------------- 1. Unit tests 2. manual tests done - agent1.sinks.log-sink1.hdfs.filePrefix = flume-data/%D results in publishing data with timestamp at agent's hop running HDFSEventSink Thanks, Inder
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-04-21 04:36:01, Mike Percy wrote:

          > flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java, line 29

          > <https://reviews.apache.org/r/4826/diff/1/?file=103469#file103469line29>

          >

          > Does this class belong in the core at all?

          >

          > In my view, this class isn't general enough to be checked into the Flume core. This should be kept as an application-specific plugin which is added to the runtime classpath and referenced in the config. That's sort of the point of adding this EventHeaderDecorator plugin interface, right?

          Mike,

          makes sense. Probably a plugins package might help. It would be nice to ship the custom plugins along with FLUME.
          I will incorporate the feedback and revert.

          • Inder

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4826/#review7102
          -----------------------------------------------------------

          On 2012-04-20 10:33:14, Inder Singh wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4826/

          -----------------------------------------------------------

          (Updated 2012-04-20 10:33:14)

          Review request for Flume, Arvind Prabhakar, Mike Percy, and Hari Shreedharan.

          Summary

          -------

          Overview of Changes

          1.EventHeaderDecorator -> generic class having processHeader()

          2.SinkEventHeaderDecorator is a EventHeaderDecorator which overrides timestamp header at this hop.

          3.EventHeaderDecoratorFactory provides EventHeaderDecorator's based on <sink>.headerDecorator config

          4.AbstractSink -> has a eventHeaderDecortaor

          5.HDFSEventSink -> checks for decorator through config and calls processHeader() for each event which modifies/add headers based on event decorator defined through config.

          Sample Configuration for testing

          ----------------------------------

          agent1.sinks.log-sink1.hdfs.headerDecorator = sinkDecorator

          This addresses bug FLUME-1097.

          https://issues.apache.org/jira/browse/FLUME-1097

          Diffs

          -----

          flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecorator.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecoratorFactory.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059

          flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 1b61cad

          Diff: https://reviews.apache.org/r/4826/diff

          Testing

          -------

          Testing Done

          -------------

          1. Unit tests

          2. manual tests done - agent1.sinks.log-sink1.hdfs.filePrefix = flume-data/%D results in publishing data with timestamp at agent's hop running HDFSEventSink

          Thanks,

          Inder

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-04-21 04:36:01, Mike Percy wrote: > flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java, line 29 > < https://reviews.apache.org/r/4826/diff/1/?file=103469#file103469line29 > > > Does this class belong in the core at all? > > In my view, this class isn't general enough to be checked into the Flume core. This should be kept as an application-specific plugin which is added to the runtime classpath and referenced in the config. That's sort of the point of adding this EventHeaderDecorator plugin interface, right? Mike, makes sense. Probably a plugins package might help. It would be nice to ship the custom plugins along with FLUME. I will incorporate the feedback and revert. Inder ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4826/#review7102 ----------------------------------------------------------- On 2012-04-20 10:33:14, Inder Singh wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4826/ ----------------------------------------------------------- (Updated 2012-04-20 10:33:14) Review request for Flume, Arvind Prabhakar, Mike Percy, and Hari Shreedharan. Summary ------- Overview of Changes 1.EventHeaderDecorator -> generic class having processHeader() 2.SinkEventHeaderDecorator is a EventHeaderDecorator which overrides timestamp header at this hop. 3.EventHeaderDecoratorFactory provides EventHeaderDecorator's based on <sink>.headerDecorator config 4.AbstractSink -> has a eventHeaderDecortaor 5.HDFSEventSink -> checks for decorator through config and calls processHeader() for each event which modifies/add headers based on event decorator defined through config. Sample Configuration for testing ---------------------------------- agent1.sinks.log-sink1.hdfs.headerDecorator = sinkDecorator This addresses bug FLUME-1097 . https://issues.apache.org/jira/browse/FLUME-1097 Diffs ----- flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecorator.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecoratorFactory.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 1b61cad Diff: https://reviews.apache.org/r/4826/diff Testing ------- Testing Done ------------- 1. Unit tests 2. manual tests done - agent1.sinks.log-sink1.hdfs.filePrefix = flume-data/%D results in publishing data with timestamp at agent's hop running HDFSEventSink Thanks, Inder
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4826/#review7413
          -----------------------------------------------------------

          Hey; On further reflection, this approach just isn't great fit. The use case makes sense. However what we need is a decorator framework, not a sink-specific solution, and that is a relatively ambitious undertaking at this point.

          Let's hold off on pursuing sink-specific decorators until a framework is put into place. I'm working on a source/sink-agnostic approach in FLUME-1157 and will provide updates on that JIRA soon.

          • Mike

          On 2012-04-20 10:33:14, Inder Singh wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4826/

          -----------------------------------------------------------

          (Updated 2012-04-20 10:33:14)

          Review request for Flume, Arvind Prabhakar, Mike Percy, and Hari Shreedharan.

          Summary

          -------

          Overview of Changes

          1.EventHeaderDecorator -> generic class having processHeader()

          2.SinkEventHeaderDecorator is a EventHeaderDecorator which overrides timestamp header at this hop.

          3.EventHeaderDecoratorFactory provides EventHeaderDecorator's based on <sink>.headerDecorator config

          4.AbstractSink -> has a eventHeaderDecortaor

          5.HDFSEventSink -> checks for decorator through config and calls processHeader() for each event which modifies/add headers based on event decorator defined through config.

          Sample Configuration for testing

          ----------------------------------

          agent1.sinks.log-sink1.hdfs.headerDecorator = sinkDecorator

          This addresses bug FLUME-1097.

          https://issues.apache.org/jira/browse/FLUME-1097

          Diffs

          -----

          flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecorator.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecoratorFactory.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059

          flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 1b61cad

          Diff: https://reviews.apache.org/r/4826/diff

          Testing

          -------

          Testing Done

          -------------

          1. Unit tests

          2. manual tests done - agent1.sinks.log-sink1.hdfs.filePrefix = flume-data/%D results in publishing data with timestamp at agent's hop running HDFSEventSink

          Thanks,

          Inder

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4826/#review7413 ----------------------------------------------------------- Hey; On further reflection, this approach just isn't great fit. The use case makes sense. However what we need is a decorator framework, not a sink-specific solution, and that is a relatively ambitious undertaking at this point. Let's hold off on pursuing sink-specific decorators until a framework is put into place. I'm working on a source/sink-agnostic approach in FLUME-1157 and will provide updates on that JIRA soon. Mike On 2012-04-20 10:33:14, Inder Singh wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4826/ ----------------------------------------------------------- (Updated 2012-04-20 10:33:14) Review request for Flume, Arvind Prabhakar, Mike Percy, and Hari Shreedharan. Summary ------- Overview of Changes 1.EventHeaderDecorator -> generic class having processHeader() 2.SinkEventHeaderDecorator is a EventHeaderDecorator which overrides timestamp header at this hop. 3.EventHeaderDecoratorFactory provides EventHeaderDecorator's based on <sink>.headerDecorator config 4.AbstractSink -> has a eventHeaderDecortaor 5.HDFSEventSink -> checks for decorator through config and calls processHeader() for each event which modifies/add headers based on event decorator defined through config. Sample Configuration for testing ---------------------------------- agent1.sinks.log-sink1.hdfs.headerDecorator = sinkDecorator This addresses bug FLUME-1097 . https://issues.apache.org/jira/browse/FLUME-1097 Diffs ----- flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecorator.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecoratorFactory.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 1b61cad Diff: https://reviews.apache.org/r/4826/diff Testing ------- Testing Done ------------- 1. Unit tests 2. manual tests done - agent1.sinks.log-sink1.hdfs.filePrefix = flume-data/%D results in publishing data with timestamp at agent's hop running HDFSEventSink Thanks, Inder
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-05-01 04:53:44, Mike Percy wrote:

          > Hey; On further reflection, this approach just isn't great fit. The use case makes sense. However what we need is a decorator framework, not a sink-specific solution, and that is a relatively ambitious undertaking at this point.

          >

          > Let's hold off on pursuing sink-specific decorators until a framework is put into place. I'm working on a source/sink-agnostic approach in FLUME-1157 and will provide updates on that JIRA soon.

          Sure Mike, will wait to get the design from the JIRA

          • Inder

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4826/#review7413
          -----------------------------------------------------------

          On 2012-04-20 10:33:14, Inder Singh wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4826/

          -----------------------------------------------------------

          (Updated 2012-04-20 10:33:14)

          Review request for Flume, Arvind Prabhakar, Mike Percy, and Hari Shreedharan.

          Summary

          -------

          Overview of Changes

          1.EventHeaderDecorator -> generic class having processHeader()

          2.SinkEventHeaderDecorator is a EventHeaderDecorator which overrides timestamp header at this hop.

          3.EventHeaderDecoratorFactory provides EventHeaderDecorator's based on <sink>.headerDecorator config

          4.AbstractSink -> has a eventHeaderDecortaor

          5.HDFSEventSink -> checks for decorator through config and calls processHeader() for each event which modifies/add headers based on event decorator defined through config.

          Sample Configuration for testing

          ----------------------------------

          agent1.sinks.log-sink1.hdfs.headerDecorator = sinkDecorator

          This addresses bug FLUME-1097.

          https://issues.apache.org/jira/browse/FLUME-1097

          Diffs

          -----

          flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecorator.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecoratorFactory.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059

          flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 1b61cad

          Diff: https://reviews.apache.org/r/4826/diff

          Testing

          -------

          Testing Done

          -------------

          1. Unit tests

          2. manual tests done - agent1.sinks.log-sink1.hdfs.filePrefix = flume-data/%D results in publishing data with timestamp at agent's hop running HDFSEventSink

          Thanks,

          Inder

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-05-01 04:53:44, Mike Percy wrote: > Hey; On further reflection, this approach just isn't great fit. The use case makes sense. However what we need is a decorator framework, not a sink-specific solution, and that is a relatively ambitious undertaking at this point. > > Let's hold off on pursuing sink-specific decorators until a framework is put into place. I'm working on a source/sink-agnostic approach in FLUME-1157 and will provide updates on that JIRA soon. Sure Mike, will wait to get the design from the JIRA Inder ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4826/#review7413 ----------------------------------------------------------- On 2012-04-20 10:33:14, Inder Singh wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4826/ ----------------------------------------------------------- (Updated 2012-04-20 10:33:14) Review request for Flume, Arvind Prabhakar, Mike Percy, and Hari Shreedharan. Summary ------- Overview of Changes 1.EventHeaderDecorator -> generic class having processHeader() 2.SinkEventHeaderDecorator is a EventHeaderDecorator which overrides timestamp header at this hop. 3.EventHeaderDecoratorFactory provides EventHeaderDecorator's based on <sink>.headerDecorator config 4.AbstractSink -> has a eventHeaderDecortaor 5.HDFSEventSink -> checks for decorator through config and calls processHeader() for each event which modifies/add headers based on event decorator defined through config. Sample Configuration for testing ---------------------------------- agent1.sinks.log-sink1.hdfs.headerDecorator = sinkDecorator This addresses bug FLUME-1097 . https://issues.apache.org/jira/browse/FLUME-1097 Diffs ----- flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecorator.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecoratorFactory.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 1b61cad Diff: https://reviews.apache.org/r/4826/diff Testing ------- Testing Done ------------- 1. Unit tests 2. manual tests done - agent1.sinks.log-sink1.hdfs.filePrefix = flume-data/%D results in publishing data with timestamp at agent's hop running HDFSEventSink Thanks, Inder
          Hide
          Mike Percy added a comment -

          If this issue has been resolved by FLUME-1215 can it please be closed?

          Show
          Mike Percy added a comment - If this issue has been resolved by FLUME-1215 can it please be closed?
          Hide
          Inder SIngh added a comment -

          Closing this as FLUME-1215 supports this requirement in a more generic way.

          Show
          Inder SIngh added a comment - Closing this as FLUME-1215 supports this requirement in a more generic way.

            People

            • Assignee:
              Inder SIngh
              Reporter:
              Inder SIngh
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development