Flume
  1. Flume
  2. FLUME-2242

Flume Sink and Source for Apache Kafka

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: v1.4.0
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
    • Environment:

      kafka, scala jar dependencies

      Description

      Source and Sinks for LinkedIn Kafka 0.7.2. I use my code in our company and runs in 100k msg/sec.

      My Project:https://github.com/baniuyao/flume-kafka

      1. Add support for Kafka Source Sub-task Open Unassigned
       
      2. Add support for Kafka Sink Sub-task Open Unassigned
       

        Activity

        Hide
        Hari Shreedharan added a comment -

        Frank Yao - Thanks for the patches! I will review this in a few days.

        Show
        Hari Shreedharan added a comment - Frank Yao - Thanks for the patches! I will review this in a few days.
        Hide
        Frank Yao added a comment -

        Patch attached. Plz review, thanks.

        Show
        Frank Yao added a comment - Patch attached. Plz review, thanks.
        Hide
        Frank Yao added a comment -

        Unitests are all ready for sinks and sources, tomorrow I'll read 'how-to-contribut' and fix bugs.

        Show
        Frank Yao added a comment - Unitests are all ready for sinks and sources, tomorrow I'll read 'how-to-contribut' and fix bugs.
        Hide
        Ashish Paliwal added a comment -

        I am not sure on this. Best would be to ask on dev ML

        Show
        Ashish Paliwal added a comment - I am not sure on this. Best would be to ask on dev ML
        Hide
        Frank Yao added a comment -

        okay. For kafka 0.7.2 maven problem, we can put kafka-0.7.2.jar in local directory, and in pom.xml we use project.local like below. Does it make sense?
        <repositories>
        <repository>
        <id>project.local</id>
        <name>project</name>
        <url>file:$

        {project.basedir}

        /extra-dependencies</url>
        </repository>
        </repositories>

        Show
        Frank Yao added a comment - okay. For kafka 0.7.2 maven problem, we can put kafka-0.7.2.jar in local directory, and in pom.xml we use project.local like below. Does it make sense? <repositories> <repository> <id>project.local</id> <name>project</name> <url> file:$ {project.basedir} /extra-dependencies</url> </repository> </repositories>
        Hide
        Ashish Paliwal added a comment -

        Great! Once done, please upload the patch for review (https://reviews.apache.org). More details here https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute

        Show
        Ashish Paliwal added a comment - Great! Once done, please upload the patch for review ( https://reviews.apache.org ). More details here https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute
        Hide
        Frank Yao added a comment -

        Source test cases are ready today. Tomorrow I will adjust coding format and add sink test cases.

        Show
        Frank Yao added a comment - Source test cases are ready today. Tomorrow I will adjust coding format and add sink test cases.
        Hide
        Ashish Paliwal added a comment -

        Great ! The problem is Kafka 0.7.2 is not available in Maven repo, so am not sure how jenkins builds would work. We might have to find a way for this to work in automated environment. I am working on Sink part, shall attach the patch as soon as I am done. If you have any suggestions on 0.7.2 version's maven integration, let me know. I shall rebase the Source back to 0.7.2

        Show
        Ashish Paliwal added a comment - Great ! The problem is Kafka 0.7.2 is not available in Maven repo, so am not sure how jenkins builds would work. We might have to find a way for this to work in automated environment. I am working on Sink part, shall attach the patch as soon as I am done. If you have any suggestions on 0.7.2 version's maven integration, let me know. I shall rebase the Source back to 0.7.2
        Hide
        Frank Yao added a comment -

        I'm working on Unit Test cases tonight.

        For Kafka version, as far as I know, 0.7.2 is more mature. 0.8 has changed a lot, and some opensource tools have not got ready to it, for example, Twitter Storm support Kafka 0.7 in its storm-contrib currently.

        I think we should make flume source&sink available for kafka 0.7.2 and then, start get kafka 0.8 ready.

        Show
        Frank Yao added a comment - I'm working on Unit Test cases tonight. For Kafka version, as far as I know, 0.7.2 is more mature. 0.8 has changed a lot, and some opensource tools have not got ready to it, for example, Twitter Storm support Kafka 0.7 in its storm-contrib currently. I think we should make flume source&sink available for kafka 0.7.2 and then, start get kafka 0.8 ready.
        Hide
        Ashish Paliwal added a comment -

        I added sub-tasks for Source and Sink, as they shall go in separate packages. Kafka 0.7.2 is not in Maven, do you think we should upgrade to 0.8 beta version?

        Show
        Ashish Paliwal added a comment - I added sub-tasks for Source and Sink, as they shall go in separate packages. Kafka 0.7.2 is not in Maven, do you think we should upgrade to 0.8 beta version?
        Hide
        Ashish Paliwal added a comment -

        Any progress with it? I am planning to work on it sometime today and add the work to this JIRA for further refinement. Will take a while to submit the patch for review.

        Show
        Ashish Paliwal added a comment - Any progress with it? I am planning to work on it sometime today and add the work to this JIRA for further refinement. Will take a while to submit the patch for review.
        Hide
        Frank Yao added a comment -

        Okay, I would work on my part ASAP.

        Show
        Frank Yao added a comment - Okay, I would work on my part ASAP.
        Hide
        Ashish Paliwal added a comment -

        No worries, we are closer in timezone, just an hour apart.

        Meanwhile, we will have more on the License stuff by then.

        Show
        Ashish Paliwal added a comment - No worries, we are closer in timezone, just an hour apart. Meanwhile, we will have more on the License stuff by then.
        Hide
        Frank Yao added a comment -

        Thanks, I'll do these tomorrow since it is 9 p.m in Shanghai.

        Show
        Frank Yao added a comment - Thanks , I'll do these tomorrow since it is 9 p.m in Shanghai.
        Hide
        Ashish Paliwal added a comment -

        AFAIK, you need to sign a ICLA. More here http://apache.org/licenses/ see section "Contributor License Agreements". I am not the expert on Licenses part and would let more experienced folks chime in for that.

        On other side, would be great if you can create a patch, attach to JIRA and open a review request. Once review is completed one of the committers can commit the code. More info on this https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute

        This would be a good addition to Flume.

        Few things that we would need to do are
        1. Update the pkg structure and add ASL header
        2. Add Unit Test cases

        Show
        Ashish Paliwal added a comment - AFAIK, you need to sign a ICLA. More here http://apache.org/licenses/ see section "Contributor License Agreements". I am not the expert on Licenses part and would let more experienced folks chime in for that. On other side, would be great if you can create a patch, attach to JIRA and open a review request. Once review is completed one of the committers can commit the code. More info on this https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute This would be a good addition to Flume. Few things that we would need to do are 1. Update the pkg structure and add ASL header 2. Add Unit Test cases
        Hide
        Frank Yao added a comment -

        Paliwal: there is not any licences in this project yet. Should I add ASL v2 in that?

        Show
        Frank Yao added a comment - Paliwal: there is not any licences in this project yet. Should I add ASL v2 in that?
        Hide
        Ashish Paliwal added a comment -

        Is the project under ASL v2?

        Show
        Ashish Paliwal added a comment - Is the project under ASL v2?
        Hide
        Ashish Paliwal added a comment -

        Updated Affected Release version to 1.4.0 as 1.5.0 is not release yet

        Show
        Ashish Paliwal added a comment - Updated Affected Release version to 1.4.0 as 1.5.0 is not release yet

          People

          • Assignee:
            Unassigned
            Reporter:
            Frank Yao
          • Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development