[FLUME-2499] Include Kafka Message Key in Event Header, Updated Comments

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.5.0.1
• Fix Version/s: 1.6.0
• Component/s: Sinks+Sources
• Assignee: Ricky Saltzer
• Reporter: Ricky Saltzer
• Created: 10/Oct/14 21:13
• Updated: 17/Apr/15 01:21
• Resolved: 16/Oct/14 05:22

Description

I believe it would be useful to include a message's key in the event header, as users will often want to partition based on this key when ingesting into a sink.

Example:

sfdc.sinks.hdfs.hdfs.path = /data/testing/kafka_flume/dataset=%{key}/year=%Y/month=%m/day=%d/

In this case, the first part of the partition key is the Kafka message key, which will ensure messages are organized by key.

I also updated some of the comments and cleaned up the code slightly.

Cheers,
Ricky