Updated Affected Release version to 1.4.0 as 1.5.0 is not release yet

Is the project under ASL v2?

Paliwal: there is not any licences in this project yet. Should I add ASL v2 in that?

AFAIK, you need to sign a ICLA. More here http://apache.org/licenses/ see section "Contributor License Agreements". I am not the expert on Licenses part and would let more experienced folks chime in for that.

On other side, would be great if you can create a patch, attach to JIRA and open a review request. Once review is completed one of the committers can commit the code. More info on this https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute

This would be a good addition to Flume.

Few things that we would need to do are
1. Update the pkg structure and add ASL header
2. Add Unit Test cases

Thanks, I'll do these tomorrow since it is 9 p.m in Shanghai.

No worries, we are closer in timezone, just an hour apart.

Meanwhile, we will have more on the License stuff by then.

Okay, I would work on my part ASAP.

Any progress with it? I am planning to work on it sometime today and add the work to this JIRA for further refinement. Will take a while to submit the patch for review.

I added sub-tasks for Source and Sink, as they shall go in separate packages. Kafka 0.7.2 is not in Maven, do you think we should upgrade to 0.8 beta version?

I'm working on Unit Test cases tonight.

For Kafka version, as far as I know, 0.7.2 is more mature. 0.8 has changed a lot, and some opensource tools have not got ready to it, for example, Twitter Storm support Kafka 0.7 in its storm-contrib currently.

I think we should make flume source&sink available for kafka 0.7.2 and then, start get kafka 0.8 ready.

Great ! The problem is Kafka 0.7.2 is not available in Maven repo, so am not sure how jenkins builds would work. We might have to find a way for this to work in automated environment. I am working on Sink part, shall attach the patch as soon as I am done. If you have any suggestions on 0.7.2 version's maven integration, let me know. I shall rebase the Source back to 0.7.2

Source test cases are ready today. Tomorrow I will adjust coding format and add sink test cases.

Great! Once done, please upload the patch for review (https://reviews.apache.org). More details here https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute

okay. For kafka 0.7.2 maven problem, we can put kafka-0.7.2.jar in local directory, and in pom.xml we use project.local like below. Does it make sense?
<repositories>
<repository>
<id>project.local</id>
<name>project</name>
<url>file:$

/extra-dependencies</url>
</repository>
</repositories>

I am not sure on this. Best would be to ask on dev ML

Unitests are all ready for sinks and sources, tomorrow I'll read 'how-to-contribut' and fix bugs.

Patch attached. Plz review, thanks.

Frank Yao - Thanks for the patches! I will review this in a few days.

Hi,

Any update on this patch ? kafka 0.8 has been here since late 2013 for now. It would be chump to release Flume 1.5.0 with kafka 7.2 support to me.

Is there someone working on kafka 0.8+ source/sink ? If I can help...

I am looking to use kafka 0.8 sink with flume-ng 1.5. Will appreciate any updates on the integration progress and any help needed in testing etc.

This is a Flume-Kafka sink implementation for Kafka 0.8.1.1 and Flume 1.5.0.

https://github.com/thilinamb/flume-ng-kafka-sink

Hope this helps.

-Thilina

Thanks @Thilina

Hope the code is rolled into trunk

We committed it to trunk. It will be in the 1.6 release of Flume.

Thanks. The kafka source/sink are much awaited pieces of integration.