[FLUME-1045] Proposal to support disk based spooling

• Type: New Feature
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.0.0
• Labels: patch
• Release Note: I
• Assignee: Unassigned
• Reporter: Inder SIngh
• Created: 22/Mar/12 09:29
• Updated: 28/Feb/14 17:36

Description

1. Problem Description
A sink being unavailable at any stage in the pipeline causes it to back-off and retry after a while. Channel's associated with such sinks start buffering data with the caveat that if you are using a memory channel it can result in a domino effect on the entire pipeline. There could be legitimate down times eg: HDFS sink being down for name node maintenance, hadoop upgrades.

2. Why not use a durable channel (JDBC, FileChannel)?
Want high throughput and support sink down times as a first class use-case.

Issue Links

• relates to: FLUME-1227 Introduce some sort of SpillableChannel