[FLUME-1201] Create a buffer channel, that stores overflow from a fast, low capacity channel to a slower high capacity channel

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Component/s: Channel
• Assignee: Juhani Connolly
• Reporter: Juhani Connolly
• Created: 14/May/12 03:57
• Updated: 09/Jul/12 22:23
• Resolved: 29/May/12 00:26

Description

As it stands, users need to make a choice between either slower channels or faster ones. The current "middle ground" consists of RecoverableFileChannel which still stores everything to disk. MemoryChannel on the other hand has limited capacity or a high memory footprint.
I propose to implement a buffer channel, somewhat like the buffer store in scribed.
It would normally act as a proxy to the primary channel. Should this channel be unable to receive events(normally because it is at capacity, but perhaps some future channels may have other failure cases) it would switch states to buffering, storing new events to the secondary channel.
In buffering, items continue to be read from the primary channel, and it attempts to "refill" itself from the secondary. Once the secondary is found to be empty, operation switches back to streaming mode, with items going directly to the primary.

The main objective of this would thus be to have a high throughput channel as the primary mode of operation, allowing it to switch over when takes are not keeping up with puts.