The core requirement here is that not all the metadata transformed / transferred to the FlumeEvent. So that interceptors and sinks can use the data.
This is a blocking issue for the organisation I work for to implement flume in our workflows where we use Kafka. (I am also aware of another org).
RE The Solution:
The intent was to split consumption from transformation into separate entities, it keeps the two concerns separate.
The design actually as per ticket, mimics that of the more mature JMS solution already in Flume. Its intent is not as an interceptor but to convert from source object event to FlumeEvent.
Happy to merge the two, but I think it is cleaner to separate the transform from source system event to flume event and the consume code, and I think the solution in JMS in cleaner and neater.
As for the number of headers this is far few headers / meta data than will come from sources such as JMS. Also to note, the data will already will be in memory as in MessageAndMetaData that we transform from.
RE Support for Kafka 0.9.0
This ticket has been open since sept, as patch is developed, and solves the issue and all existing tests pass I do not see any reason to hold off merging the functionality. Unless the new consumer is ready for next release with these features supported I do not see why one would hold off.
Also need to be aware kafka 0.9.0 consumer has only just released and as such not many organisations will upgrade instantly, as such using the old consumer for flume for some period of time will be needed for compatibility. As the new consumer in 0.9.0 is not back compatible with older kafka brokers. But older consumers are compatible with the 0.9.0 broker.