Flume
  1. Flume
  2. FLUME-2140

Support diverting bad events from pipeline

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Node
    • Labels:
      None

      Description

      A bad event can be any event that causes persistent sink side processing failure due to the inherent nature of the event itself. Note that failures that are not related to the inherent nature of the event such as network communication failure, downstream capacity failure etc., do not make the event a bad-event.

      The presence of a bad event in a channel can cause the entire pipleline to choke and become unusable. Flume should therefore be able to identify bad events and provide a facility to route them out of the pipleline in order to ensure the transport of other events continues uninterrupted.

        Activity

        Hide
        Roshan Naik added a comment -

        It appears that may be only the sink is able to know for certain (at the time to delivery) that a particular event is bad.

        One scenario that comes to mind here:

        • an event that may be bad for one sink maynot be bad for another. for instance, logger sink or rolling file sink may not have trouble delivering an event that is troublesome for elastic search sink.
        Show
        Roshan Naik added a comment - It appears that may be only the sink is able to know for certain (at the time to delivery) that a particular event is bad. One scenario that comes to mind here: an event that may be bad for one sink maynot be bad for another. for instance, logger sink or rolling file sink may not have trouble delivering an event that is troublesome for elastic search sink.
        Hide
        Arvind Prabhakar added a comment -

        Another case - a downstream filter is buggy and causes a batch to fail repeatedly due to a malformed header or some other details.

        Show
        Arvind Prabhakar added a comment - Another case - a downstream filter is buggy and causes a batch to fail repeatedly due to a malformed header or some other details.
        Hide
        Arvind Prabhakar added a comment -

        Discussion thread on this subject in the user-list for reference.

        Show
        Arvind Prabhakar added a comment - Discussion thread on this subject in the user-list for reference.
        Hide
        Jean lagarutte added a comment -

        hello
        any news or advance in this problem ?
        This is a major stopper for me and i'm currently considering switching to another software.

        The events comes from several sources and sometimes they are malformed.
        But the regex interceptor can still parse them but generate some headers with special characters.
        The hdfs sink use theses header for the path but with characters like '\ * or :' it fails to deliver event.

        I need to restart flume and i lost all event frequently.
        The failover sink is not really a good option for me.

        Can't we just drop the bad event and write them to agent logfile ?

        Show
        Jean lagarutte added a comment - hello any news or advance in this problem ? This is a major stopper for me and i'm currently considering switching to another software. The events comes from several sources and sometimes they are malformed. But the regex interceptor can still parse them but generate some headers with special characters. The hdfs sink use theses header for the path but with characters like '\ * or :' it fails to deliver event. I need to restart flume and i lost all event frequently. The failover sink is not really a good option for me. Can't we just drop the bad event and write them to agent logfile ?

          People

          • Assignee:
            Unassigned
            Reporter:
            Arvind Prabhakar
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development