Kafka
  1. Kafka
  2. KAFKA-1326 New consumer checklist
  3. KAFKA-1330

Implement subscribe(TopicPartition...partitions) and pool(timeout) in the consumer

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.9.0
    • Fix Version/s: None
    • Component/s: consumer
    • Labels:
      None

      Description

      This involves adding basic fetch functionality (equivalent to SimpleConsumer) to the new consumer. Effectively:

      1) Implement subscribe(TopicPartition...partitions) and unsubscribe(TopicPartition...partitions).
      2) Implement pool(timeout).
      3) Add memory management to the consumer.

      1. KAFKA-1330_2014-07-21_16:44:23.patch
        162 kB
        Jay Kreps
      2. KAFKA-1330.patch
        177 kB
        Jay Kreps

        Issue Links

          Activity

          Hide
          nicu marasoiu added a comment -

          Hi,

          If I want to take this, how much time to get up to speed do I have?
          Which are the documents to read?

          I have kafka experience and scala courses, I will try to jump in

          Show
          nicu marasoiu added a comment - Hi, If I want to take this, how much time to get up to speed do I have? Which are the documents to read? I have kafka experience and scala courses, I will try to jump in
          Hide
          Jay Kreps added a comment -

          Created reviewboard https://reviews.apache.org/r/23442/
          against branch trunk

          Show
          Jay Kreps added a comment - Created reviewboard https://reviews.apache.org/r/23442/ against branch trunk
          Hide
          Jay Kreps added a comment -

          Hey nicu marasoiu, sorry had actually been working on this for the last week...I should have assigned it to myself.

          Show
          Jay Kreps added a comment - Hey nicu marasoiu , sorry had actually been working on this for the last week...I should have assigned it to myself.
          Hide
          Jay Kreps added a comment -

          Guys, I posted a very rough draft that implements "simple consumer" functionality. I think implementing offset maintainence on top of this will be pretty easy and can be done once we have those new request defn's done. I'd like to start getting these changes in even though they aren't done to avoid keeping large patches against trunk. Specifically what I am hoping is that we can do a detailed review on the changes to common code, esp with respect to NetworkClient and be a bit lax on KafkaConsumer until we get something more complete.

          Show
          Jay Kreps added a comment - Guys, I posted a very rough draft that implements "simple consumer" functionality. I think implementing offset maintainence on top of this will be pretty easy and can be done once we have those new request defn's done. I'd like to start getting these changes in even though they aren't done to avoid keeping large patches against trunk. Specifically what I am hoping is that we can do a detailed review on the changes to common code, esp with respect to NetworkClient and be a bit lax on KafkaConsumer until we get something more complete.
          Hide
          Jay Kreps added a comment -

          Updated reviewboard https://reviews.apache.org/r/23442/
          against branch trunk

          Show
          Jay Kreps added a comment - Updated reviewboard https://reviews.apache.org/r/23442/ against branch trunk
          Hide
          Jay Kreps added a comment -

          Done as part of KAFKA-1760

          Show
          Jay Kreps added a comment - Done as part of KAFKA-1760

            People

            • Assignee:
              Jay Kreps
              Reporter:
              Neha Narkhede
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development