Flume
  1. Flume
  2. FLUME-1947

ExecSource should optionally block if the channel is full.

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      This is so that the process executed blocks till the channel clears up. This is exceptionally useful if the process is a tail -F or something like that.

      In addition, the exec source does have issues with respect to the last batch not being sent. We could also specify a timeout at the end of which the batch should be flushed to the channel if there are no new events.

      1. FLUME-1947-1.patch
        9 kB
        Hari Shreedharan
      2. FLUME-1947.patch
        15 kB
        Hari Shreedharan

        Activity

        Hide
        Yongkun Wang added a comment -

        This would be a nice feature otherwise I have to write a custom source to do blocking.

        Show
        Yongkun Wang added a comment - This would be a nice feature otherwise I have to write a custom source to do blocking.
        Hide
        Hari Shreedharan added a comment -

        bump.

        Show
        Hari Shreedharan added a comment - bump.
        Hide
        Mike Percy added a comment -

        Hari, apologies, but this one seems a bit risky to commit the day of rolling an RC. I'm leaning toward not including it in 1.4.0 unless you feel that it's low risk and want to include it. Please let me know.

        Show
        Mike Percy added a comment - Hari, apologies, but this one seems a bit risky to commit the day of rolling an RC. I'm leaning toward not including it in 1.4.0 unless you feel that it's low risk and want to include it. Please let me know.
        Hide
        Hari Shreedharan added a comment -

        Looks like some of the code from this patch went into another one. I will take a look at this soon.

        Show
        Hari Shreedharan added a comment - Looks like some of the code from this patch went into another one. I will take a look at this soon.
        Hide
        Mike Percy added a comment -

        I'd like to get this into Flume 1.4, if you have time to rebase that would be great (trying git merge didn't work for me). If not we'll push this back to a future release.

        Show
        Mike Percy added a comment - I'd like to get this into Flume 1.4, if you have time to rebase that would be great (trying git merge didn't work for me). If not we'll push this back to a future release.
        Hide
        Hari Shreedharan added a comment -

        Thanks Mike. Yes, this patch has an implementation to push the events into the channel based on timeout (which got committed in FLUME-1819). Removing that should fix this patch. I will attach a new patch which removes that.

        Show
        Hari Shreedharan added a comment - Thanks Mike. Yes, this patch has an implementation to push the events into the channel based on timeout (which got committed in FLUME-1819 ). Removing that should fix this patch. I will attach a new patch which removes that.
        Hide
        Mike Percy added a comment -

        Hari, after FLUME-1819 this patch no longer applies. Looks like a few changes have gone in and this one needs rework.

        Show
        Mike Percy added a comment - Hari, after FLUME-1819 this patch no longer applies. Looks like a few changes have gone in and this one needs rework.
        Hide
        Hari Shreedharan added a comment -

        Made multiple changes to the exec source:

        1. Flush the events to the channel, based on a configurable timeout - even when a full batch was not received.
        2. Block the spawned process until channel exceptions are cleared up by not reading the stdout buffer. Backoff exponentially (subject to an upper limit), till events are available.

        Show
        Hari Shreedharan added a comment - Made multiple changes to the exec source: 1. Flush the events to the channel, based on a configurable timeout - even when a full batch was not received. 2. Block the spawned process until channel exceptions are cleared up by not reading the stdout buffer. Backoff exponentially (subject to an upper limit), till events are available.

          People

          • Assignee:
            Hari Shreedharan
            Reporter:
            Hari Shreedharan
          • Votes:
            3 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development