Flume
  1. Flume
  2. FLUME-2426

Support interceptors in the Embedded Agent

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.5.0.1
    • Fix Version/s: v1.6.0
    • Component/s: None

      Description

      Interceptors are currently not supported in Embedded Agents. Interceptor seems to be a critical feature as it is the only way to add custom headers to a Flume Event. Custom headers are required for other features such as channel selection and dynamic filenames (using %

      {<header>}

      )

        Activity

        Hide
        Hari Shreedharan added a comment -

        You could add the header before passing the events to the agent no? I don't see what additional info you'd have in the embedded agent which you would not have in your own application.

        Show
        Hari Shreedharan added a comment - You could add the header before passing the events to the agent no? I don't see what additional info you'd have in the embedded agent which you would not have in your own application.
        Hide
        Bijith Kumar added a comment -

        You are right . However, interceptors are the only way to add a header in some scenarios like log4j embedded appenders where we don't have direct access to embedded agent instance.

        Show
        Bijith Kumar added a comment - You are right . However, interceptors are the only way to add a header in some scenarios like log4j embedded appenders where we don't have direct access to embedded agent instance.
        Hide
        Hari Shreedharan added a comment -

        Since we do not have one in the Apache Flume project, I didn't think of that. You are right though. Also if you have an implementation of a log4j appender with an embedded agent, please feel free to contribute it

        Show
        Hari Shreedharan added a comment - Since we do not have one in the Apache Flume project, I didn't think of that. You are right though. Also if you have an implementation of a log4j appender with an embedded agent, please feel free to contribute it
        Hide
        Ralph Goers added a comment -

        For reference, Bijith is referring to the Flume Appender in Log4j 2 - see LOG4J-720.

        Show
        Ralph Goers added a comment - For reference, Bijith is referring to the Flume Appender in Log4j 2 - see LOG4J-720.
        Hide
        Hari Shreedharan added a comment -

        Thanks Ralph!

        Show
        Hari Shreedharan added a comment - Thanks Ralph!
        Hide
        Bijith Kumar added a comment -

        Thanks Ralph.

        Show
        Bijith Kumar added a comment - Thanks Ralph.
        Hide
        Johny Rufus added a comment -

        I have attached a patch, that supports interceptors in the embedded agent. Except for a small change to proactively prevent NPE, rest of the changes are to the developer guide documentation and unit test cases to verify the interceptors working on the embedded agent.

        Show
        Johny Rufus added a comment - I have attached a patch, that supports interceptors in the embedded agent. Except for a small change to proactively prevent NPE, rest of the changes are to the developer guide documentation and unit test cases to verify the interceptors working on the embedded agent.
        Hide
        Hari Shreedharan added a comment -

        +1. Committing.

        Show
        Hari Shreedharan added a comment - +1. Committing.
        Hide
        ASF subversion and git services added a comment -

        Commit 59f0b4df97231acdc0b9769dccb3211c502b36d3 in flume's branch refs/heads/trunk from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=59f0b4d ]

        FLUME-2426. Support interceptors in the Embedded Agent

        (Johny Rufus via Hari)

        Show
        ASF subversion and git services added a comment - Commit 59f0b4df97231acdc0b9769dccb3211c502b36d3 in flume's branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=59f0b4d ] FLUME-2426 . Support interceptors in the Embedded Agent (Johny Rufus via Hari)
        Hide
        ASF subversion and git services added a comment -

        Commit c458a838ad3dc68946662050ee96353d36ee4e74 in flume's branch refs/heads/flume-1.6 from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=c458a83 ]

        FLUME-2426. Support interceptors in the Embedded Agent

        (Johny Rufus via Hari)

        Show
        ASF subversion and git services added a comment - Commit c458a838ad3dc68946662050ee96353d36ee4e74 in flume's branch refs/heads/flume-1.6 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=c458a83 ] FLUME-2426 . Support interceptors in the Embedded Agent (Johny Rufus via Hari)
        Hide
        Hari Shreedharan added a comment -

        Committed. Thanks Johny!

        Show
        Hari Shreedharan added a comment - Committed. Thanks Johny!
        Hide
        Hudson added a comment -

        UNSTABLE: Integrated in flume-trunk #655 (See https://builds.apache.org/job/flume-trunk/655/)
        FLUME-2426. Support interceptors in the Embedded Agent (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=59f0b4df97231acdc0b9769dccb3211c502b36d3)

        • flume-ng-embedded-agent/src/test/java/org/apache/flume/agent/embedded/TestEmbeddedAgentConfiguration.java
        • flume-ng-doc/sphinx/FlumeDeveloperGuide.rst
        • flume-ng-embedded-agent/src/test/java/org/apache/flume/agent/embedded/TestEmbeddedAgent.java
        • flume-ng-embedded-agent/src/main/java/org/apache/flume/agent/embedded/EmbeddedAgent.java
        Show
        Hudson added a comment - UNSTABLE: Integrated in flume-trunk #655 (See https://builds.apache.org/job/flume-trunk/655/ ) FLUME-2426 . Support interceptors in the Embedded Agent (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=59f0b4df97231acdc0b9769dccb3211c502b36d3 ) flume-ng-embedded-agent/src/test/java/org/apache/flume/agent/embedded/TestEmbeddedAgentConfiguration.java flume-ng-doc/sphinx/FlumeDeveloperGuide.rst flume-ng-embedded-agent/src/test/java/org/apache/flume/agent/embedded/TestEmbeddedAgent.java flume-ng-embedded-agent/src/main/java/org/apache/flume/agent/embedded/EmbeddedAgent.java
        Hide
        Hudson added a comment -

        UNSTABLE: Integrated in Flume-trunk-hbase-98 #15 (See https://builds.apache.org/job/Flume-trunk-hbase-98/15/)
        FLUME-2426. Support interceptors in the Embedded Agent (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=59f0b4df97231acdc0b9769dccb3211c502b36d3)

        • flume-ng-embedded-agent/src/test/java/org/apache/flume/agent/embedded/TestEmbeddedAgentConfiguration.java
        • flume-ng-embedded-agent/src/test/java/org/apache/flume/agent/embedded/TestEmbeddedAgent.java
        • flume-ng-embedded-agent/src/main/java/org/apache/flume/agent/embedded/EmbeddedAgent.java
        • flume-ng-doc/sphinx/FlumeDeveloperGuide.rst
        Show
        Hudson added a comment - UNSTABLE: Integrated in Flume-trunk-hbase-98 #15 (See https://builds.apache.org/job/Flume-trunk-hbase-98/15/ ) FLUME-2426 . Support interceptors in the Embedded Agent (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=59f0b4df97231acdc0b9769dccb3211c502b36d3 ) flume-ng-embedded-agent/src/test/java/org/apache/flume/agent/embedded/TestEmbeddedAgentConfiguration.java flume-ng-embedded-agent/src/test/java/org/apache/flume/agent/embedded/TestEmbeddedAgent.java flume-ng-embedded-agent/src/main/java/org/apache/flume/agent/embedded/EmbeddedAgent.java flume-ng-doc/sphinx/FlumeDeveloperGuide.rst

          People

          • Assignee:
            Johny Rufus
            Reporter:
            Bijith Kumar
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development