[FLUME-1150] Need a way to specify bucket in HDFSEventSink from client

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: Sinks+Sources
• Assignee: Unassigned
• Reporter: Mubarak Seyed
• Created: 26/Apr/12 00:01
• Updated: 17/Jun/12 06:44
• Resolved: 17/Jun/12 06:44

Description

We need a way to specify the bucket when event is streamed from client.

For instance, if i have 5 different log categories (like category [1-5]), if i use single source->channel->hdfsSink then all the logs goes to one specified directory in HDFS (/logs/<destination>) but we need something like

/logs/category1
/category2
/category3
/category4
/category5

I can use multiplexing (1 source, N channel, and N sinks) but i need to use a multiplexing channel processor to route the events(categories) to the proper sink.

What i am thinking is if i specify the category name in header in Event, HDFSEventSink (and its helper classes) can create a bucket based on header value (and then escape-sequence needs to be honored).