Flume
  1. Flume
  2. FLUME-1947

ExecSource should optionally block if the channel is full.

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      This is so that the process executed blocks till the channel clears up. This is exceptionally useful if the process is a tail -F or something like that.

      In addition, the exec source does have issues with respect to the last batch not being sent. We could also specify a timeout at the end of which the batch should be flushed to the channel if there are no new events.

      1. FLUME-1947.patch
        15 kB
        Hari Shreedharan
      2. FLUME-1947-1.patch
        9 kB
        Hari Shreedharan

        Activity

        Hari Shreedharan created issue -
        Hide
        Hari Shreedharan added a comment -

        Made multiple changes to the exec source:

        1. Flush the events to the channel, based on a configurable timeout - even when a full batch was not received.
        2. Block the spawned process until channel exceptions are cleared up by not reading the stdout buffer. Backoff exponentially (subject to an upper limit), till events are available.

        Show
        Hari Shreedharan added a comment - Made multiple changes to the exec source: 1. Flush the events to the channel, based on a configurable timeout - even when a full batch was not received. 2. Block the spawned process until channel exceptions are cleared up by not reading the stdout buffer. Backoff exponentially (subject to an upper limit), till events are available.
        Hari Shreedharan made changes -
        Field Original Value New Value
        Attachment FLUME-1947.patch [ 12573633 ]
        Hari Shreedharan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Assignee Hari Shreedharan [ hshreedharan ]
        Hide
        Mike Percy added a comment -

        Hari, after FLUME-1819 this patch no longer applies. Looks like a few changes have gone in and this one needs rework.

        Show
        Mike Percy added a comment - Hari, after FLUME-1819 this patch no longer applies. Looks like a few changes have gone in and this one needs rework.
        Hide
        Hari Shreedharan added a comment -

        Thanks Mike. Yes, this patch has an implementation to push the events into the channel based on timeout (which got committed in FLUME-1819). Removing that should fix this patch. I will attach a new patch which removes that.

        Show
        Hari Shreedharan added a comment - Thanks Mike. Yes, this patch has an implementation to push the events into the channel based on timeout (which got committed in FLUME-1819 ). Removing that should fix this patch. I will attach a new patch which removes that.
        Hide
        Mike Percy added a comment -

        I'd like to get this into Flume 1.4, if you have time to rebase that would be great (trying git merge didn't work for me). If not we'll push this back to a future release.

        Show
        Mike Percy added a comment - I'd like to get this into Flume 1.4, if you have time to rebase that would be great (trying git merge didn't work for me). If not we'll push this back to a future release.
        Hide
        Hari Shreedharan added a comment -

        Looks like some of the code from this patch went into another one. I will take a look at this soon.

        Show
        Hari Shreedharan added a comment - Looks like some of the code from this patch went into another one. I will take a look at this soon.
        Hari Shreedharan made changes -
        Attachment FLUME-1947-1.patch [ 12588439 ]
        Mike Percy made changes -
        Fix Version/s v1.4.0 [ 12323372 ]
        Hide
        Mike Percy added a comment -

        Hari, apologies, but this one seems a bit risky to commit the day of rolling an RC. I'm leaning toward not including it in 1.4.0 unless you feel that it's low risk and want to include it. Please let me know.

        Show
        Mike Percy added a comment - Hari, apologies, but this one seems a bit risky to commit the day of rolling an RC. I'm leaning toward not including it in 1.4.0 unless you feel that it's low risk and want to include it. Please let me know.
        Mike Percy made changes -
        Fix Version/s v1.4.0 [ 12323372 ]
        Hide
        Hari Shreedharan added a comment -

        bump.

        Show
        Hari Shreedharan added a comment - bump.
        Hide
        Yongkun Wang added a comment -

        This would be a nice feature otherwise I have to write a custom source to do blocking.

        Show
        Yongkun Wang added a comment - This would be a nice feature otherwise I have to write a custom source to do blocking.
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        41d 49m 1 Hari Shreedharan 23/Apr/13 06:56

          People

          • Assignee:
            Hari Shreedharan
            Reporter:
            Hari Shreedharan
          • Votes:
            3 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development