Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2242

Flume Sink and Source for Apache Kafka

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.6.0
    • Component/s: Sinks+Sources
    • Labels:
    • Environment:

      kafka, scala jar dependencies

      Description

      Source and Sinks for LinkedIn Kafka 0.7.2. I use my code in our company and runs in 100k msg/sec.

      My Project:https://github.com/baniuyao/flume-kafka

        Activity

        Hide
        xnag Xuri Nagarin added a comment -

        Thanks. The kafka source/sink are much awaited pieces of integration.

        Show
        xnag Xuri Nagarin added a comment - Thanks. The kafka source/sink are much awaited pieces of integration.
        Hide
        hshreedharan Hari Shreedharan added a comment -

        We committed it to trunk. It will be in the 1.6 release of Flume.

        Show
        hshreedharan Hari Shreedharan added a comment - We committed it to trunk. It will be in the 1.6 release of Flume.
        Hide
        xnag Xuri Nagarin added a comment -

        Thanks @Thilina

        Hope the code is rolled into trunk

        Show
        xnag Xuri Nagarin added a comment - Thanks @Thilina Hope the code is rolled into trunk
        Hide
        thilinamb Thilina Buddhika added a comment - - edited

        This is a Flume-Kafka sink implementation for Kafka 0.8.1.1 and Flume 1.5.0.

        https://github.com/thilinamb/flume-ng-kafka-sink

        Hope this helps.

        -Thilina

        Show
        thilinamb Thilina Buddhika added a comment - - edited This is a Flume-Kafka sink implementation for Kafka 0.8.1.1 and Flume 1.5.0. https://github.com/thilinamb/flume-ng-kafka-sink Hope this helps. -Thilina
        Hide
        xnag Xuri Nagarin added a comment -

        I am looking to use kafka 0.8 sink with flume-ng 1.5. Will appreciate any updates on the integration progress and any help needed in testing etc.

        Show
        xnag Xuri Nagarin added a comment - I am looking to use kafka 0.8 sink with flume-ng 1.5. Will appreciate any updates on the integration progress and any help needed in testing etc.
        Hide
        StevenLeRoux Steven Le Roux added a comment - - edited

        Hi,

        Any update on this patch ? kafka 0.8 has been here since late 2013 for now. It would be chump to release Flume 1.5.0 with kafka 7.2 support to me.

        Is there someone working on kafka 0.8+ source/sink ? If I can help...

        Show
        StevenLeRoux Steven Le Roux added a comment - - edited Hi, Any update on this patch ? kafka 0.8 has been here since late 2013 for now. It would be chump to release Flume 1.5.0 with kafka 7.2 support to me. Is there someone working on kafka 0.8+ source/sink ? If I can help...
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Frank Yao - Thanks for the patches! I will review this in a few days.

        Show
        hshreedharan Hari Shreedharan added a comment - Frank Yao - Thanks for the patches! I will review this in a few days.
        Hide
        ybaniu Frank Yao added a comment -

        Patch attached. Plz review, thanks.

        Show
        ybaniu Frank Yao added a comment - Patch attached. Plz review, thanks.
        Hide
        ybaniu Frank Yao added a comment -

        Unitests are all ready for sinks and sources, tomorrow I'll read 'how-to-contribut' and fix bugs.

        Show
        ybaniu Frank Yao added a comment - Unitests are all ready for sinks and sources, tomorrow I'll read 'how-to-contribut' and fix bugs.
        Hide
        paliwalashish Ashish Paliwal added a comment -

        I am not sure on this. Best would be to ask on dev ML

        Show
        paliwalashish Ashish Paliwal added a comment - I am not sure on this. Best would be to ask on dev ML
        Hide
        ybaniu Frank Yao added a comment -

        okay. For kafka 0.7.2 maven problem, we can put kafka-0.7.2.jar in local directory, and in pom.xml we use project.local like below. Does it make sense?
        <repositories>
        <repository>
        <id>project.local</id>
        <name>project</name>
        <url>file:$

        {project.basedir}

        /extra-dependencies</url>
        </repository>
        </repositories>

        Show
        ybaniu Frank Yao added a comment - okay. For kafka 0.7.2 maven problem, we can put kafka-0.7.2.jar in local directory, and in pom.xml we use project.local like below. Does it make sense? <repositories> <repository> <id>project.local</id> <name>project</name> <url> file:$ {project.basedir} /extra-dependencies</url> </repository> </repositories>
        Hide
        paliwalashish Ashish Paliwal added a comment -

        Great! Once done, please upload the patch for review (https://reviews.apache.org). More details here https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute

        Show
        paliwalashish Ashish Paliwal added a comment - Great! Once done, please upload the patch for review ( https://reviews.apache.org ). More details here https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute
        Hide
        ybaniu Frank Yao added a comment -

        Source test cases are ready today. Tomorrow I will adjust coding format and add sink test cases.

        Show
        ybaniu Frank Yao added a comment - Source test cases are ready today. Tomorrow I will adjust coding format and add sink test cases.
        Hide
        paliwalashish Ashish Paliwal added a comment -

        Great ! The problem is Kafka 0.7.2 is not available in Maven repo, so am not sure how jenkins builds would work. We might have to find a way for this to work in automated environment. I am working on Sink part, shall attach the patch as soon as I am done. If you have any suggestions on 0.7.2 version's maven integration, let me know. I shall rebase the Source back to 0.7.2

        Show
        paliwalashish Ashish Paliwal added a comment - Great ! The problem is Kafka 0.7.2 is not available in Maven repo, so am not sure how jenkins builds would work. We might have to find a way for this to work in automated environment. I am working on Sink part, shall attach the patch as soon as I am done. If you have any suggestions on 0.7.2 version's maven integration, let me know. I shall rebase the Source back to 0.7.2
        Hide
        ybaniu Frank Yao added a comment -

        I'm working on Unit Test cases tonight.

        For Kafka version, as far as I know, 0.7.2 is more mature. 0.8 has changed a lot, and some opensource tools have not got ready to it, for example, Twitter Storm support Kafka 0.7 in its storm-contrib currently.

        I think we should make flume source&sink available for kafka 0.7.2 and then, start get kafka 0.8 ready.

        Show
        ybaniu Frank Yao added a comment - I'm working on Unit Test cases tonight. For Kafka version, as far as I know, 0.7.2 is more mature. 0.8 has changed a lot, and some opensource tools have not got ready to it, for example, Twitter Storm support Kafka 0.7 in its storm-contrib currently. I think we should make flume source&sink available for kafka 0.7.2 and then, start get kafka 0.8 ready.
        Hide
        paliwalashish Ashish Paliwal added a comment -

        I added sub-tasks for Source and Sink, as they shall go in separate packages. Kafka 0.7.2 is not in Maven, do you think we should upgrade to 0.8 beta version?

        Show
        paliwalashish Ashish Paliwal added a comment - I added sub-tasks for Source and Sink, as they shall go in separate packages. Kafka 0.7.2 is not in Maven, do you think we should upgrade to 0.8 beta version?
        Hide
        paliwalashish Ashish Paliwal added a comment -

        Any progress with it? I am planning to work on it sometime today and add the work to this JIRA for further refinement. Will take a while to submit the patch for review.

        Show
        paliwalashish Ashish Paliwal added a comment - Any progress with it? I am planning to work on it sometime today and add the work to this JIRA for further refinement. Will take a while to submit the patch for review.
        Hide
        ybaniu Frank Yao added a comment -

        Okay, I would work on my part ASAP.

        Show
        ybaniu Frank Yao added a comment - Okay, I would work on my part ASAP.
        Hide
        paliwalashish Ashish Paliwal added a comment -

        No worries, we are closer in timezone, just an hour apart.

        Meanwhile, we will have more on the License stuff by then.

        Show
        paliwalashish Ashish Paliwal added a comment - No worries, we are closer in timezone, just an hour apart. Meanwhile, we will have more on the License stuff by then.
        Hide
        ybaniu Frank Yao added a comment -

        Thanks, I'll do these tomorrow since it is 9 p.m in Shanghai.

        Show
        ybaniu Frank Yao added a comment - Thanks , I'll do these tomorrow since it is 9 p.m in Shanghai.
        Hide
        paliwalashish Ashish Paliwal added a comment -

        AFAIK, you need to sign a ICLA. More here http://apache.org/licenses/ see section "Contributor License Agreements". I am not the expert on Licenses part and would let more experienced folks chime in for that.

        On other side, would be great if you can create a patch, attach to JIRA and open a review request. Once review is completed one of the committers can commit the code. More info on this https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute

        This would be a good addition to Flume.

        Few things that we would need to do are
        1. Update the pkg structure and add ASL header
        2. Add Unit Test cases

        Show
        paliwalashish Ashish Paliwal added a comment - AFAIK, you need to sign a ICLA. More here http://apache.org/licenses/ see section "Contributor License Agreements". I am not the expert on Licenses part and would let more experienced folks chime in for that. On other side, would be great if you can create a patch, attach to JIRA and open a review request. Once review is completed one of the committers can commit the code. More info on this https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute This would be a good addition to Flume. Few things that we would need to do are 1. Update the pkg structure and add ASL header 2. Add Unit Test cases
        Hide
        ybaniu Frank Yao added a comment -

        Paliwal: there is not any licences in this project yet. Should I add ASL v2 in that?

        Show
        ybaniu Frank Yao added a comment - Paliwal: there is not any licences in this project yet. Should I add ASL v2 in that?
        Hide
        paliwalashish Ashish Paliwal added a comment -

        Is the project under ASL v2?

        Show
        paliwalashish Ashish Paliwal added a comment - Is the project under ASL v2?
        Hide
        paliwalashish Ashish Paliwal added a comment -

        Updated Affected Release version to 1.4.0 as 1.5.0 is not release yet

        Show
        paliwalashish Ashish Paliwal added a comment - Updated Affected Release version to 1.4.0 as 1.5.0 is not release yet

          People

          • Assignee:
            Unassigned
            Reporter:
            ybaniu Frank Yao
          • Votes:
            6 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development