Flume
  1. Flume
  2. FLUME-1045

Proposal to support disk based spooling

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: v1.0.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Release Note:
      I

      Description

      1. Problem Description
      A sink being unavailable at any stage in the pipeline causes it to back-off and retry after a while. Channel's associated with such sinks start buffering data with the caveat that if you are using a memory channel it can result in a domino effect on the entire pipeline. There could be legitimate down times eg: HDFS sink being down for name node maintenance, hadoop upgrades.

      2. Why not use a durable channel (JDBC, FileChannel)?
      Want high throughput and support sink down times as a first class use-case.

      1. FLUME-1045-1.patch
        15 kB
        Inder SIngh
      2. FLUME-1045-2.patch
        15 kB
        Inder SIngh

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Unassigned
              Reporter:
              Inder SIngh
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:

                Development