[FLUME-1191] Would like autoE2EChain to have the ability to handle different classes of agents

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.9.4
• Fix Version/s: 0.9.5
• Component/s: Sinks+Sources

Environment

CentOS

Flume build:
Flume 0.9.4-cdh3u3
Git repository https://github.com/cloudera/flume/flume-core
rev unknown
Compiled by jenkins on 20120126-1114

• Assignee: Unassigned
• Reporter: Thomas Andrews
• Created: 08/May/12 21:38
• Updated: 05/Nov/14 11:31
• Resolved: 05/Nov/14 11:31

Description

We have an agent running on each of our servers, and our servers are writing audit data to these agents. The agents are configured to use autoE2EChain for their sinks.

The problem is, out of 60+ servers, five of them generate the vast majority of the traffic, and, with this many agents configured to write to our two collectors, this means that there is a high probability (about one in three) ofautoE2EChain assigning four or five of those servers to write to one collector.

Ideally, we'd have a way to set the sink as: autoE2EChain("high-traffic") so that some 'class' of agents would be assigned to collectors separately.

Alternatively, I suppose, autoE2EChain could pass a weight in: autoE2EChain(20).

I realize most work on the NG Flume, so perhaps the terminology will be different there, but the idea is likely still applicable.