Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1045

Proposal to support disk based spooling

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 1.0.0
    • None
    • None
    • I

    Description

      1. Problem Description
      A sink being unavailable at any stage in the pipeline causes it to back-off and retry after a while. Channel's associated with such sinks start buffering data with the caveat that if you are using a memory channel it can result in a domino effect on the entire pipeline. There could be legitimate down times eg: HDFS sink being down for name node maintenance, hadoop upgrades.

      2. Why not use a durable channel (JDBC, FileChannel)?
      Want high throughput and support sink down times as a first class use-case.

      Attachments

        1. FLUME-1045-2.patch
          15 kB
          Inder SIngh
        2. FLUME-1045-1.patch
          15 kB
          Inder SIngh

        Issue Links

          Activity

            People

              Unassigned Unassigned
              inder Inder SIngh
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: