Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1045

Proposal to support disk based spooling

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Release Note:
      I

      Description

      1. Problem Description
      A sink being unavailable at any stage in the pipeline causes it to back-off and retry after a while. Channel's associated with such sinks start buffering data with the caveat that if you are using a memory channel it can result in a domino effect on the entire pipeline. There could be legitimate down times eg: HDFS sink being down for name node maintenance, hadoop upgrades.

      2. Why not use a durable channel (JDBC, FileChannel)?
      Want high throughput and support sink down times as a first class use-case.

        Attachments

        1. FLUME-1045-2.patch
          15 kB
          Inder SIngh
        2. FLUME-1045-1.patch
          15 kB
          Inder SIngh

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                inder Inder SIngh
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated: