[FLUME-299] Add support for subarguments in flume config elements

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.4
• Component/s: Sinks+Sources
• Assignee: Jonathan Hsieh
• Reporter: Jonathan Hsieh
• Created: 28/Oct/10 07:44
• Updated: 06/Aug/11 00:14
• Resolved: 24/May/11 00:58

Description

It would make sense for certain arguments to have sub-arguments.

For example, when specifiying the console("avro") sink, it would be nice to give the avro output format arguments. So we could be able to say 'console(avro("json"))' or 'console(avro("bin","gzip"))'

This would also allow roll sinks to specify different trigger policies:

'roll( timer(1000))

{ ... }' // roll every second
'roll( count(100)) { ... }

' // roll every 100 events
'roll( countTimer(100,1000))

{ ... }

' // roll every 100 events or every 1000 ms
'roll( size(1000000) )

{... }

' // roll every 1000000 bytes.

It would also allow failovers sinks to specify retry policies.

Issue Links

• blocks: FLUME-273 RollSink should expose ability to configure SizeTrigger
• blocks: FLUME-541 Time+Size based roll trigger
• blocks: FLUME-632 Support arguments in output format configuration
• is related to: FLUME-472 Replace tagger with roller specific kw arg for adding a tag.
• relates to: FLUME-133 Add support for Avro Native output files.