Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-253

Refactor the async producer to have only one queue instead of one queue per broker in a Kafka cluster

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      Today, the async producer is associated with a particular broker instance, just like the SyncProducer. The Producer maintains a producer pool of sync/async producers, one per broker. Since the producer pool creates one async producer per broker, we have multiple producer queues for one Producer instance.

      With replication, a topic partition will be logical. This requires refactoring the AsyncProducer to be broker agnostic. As a side effect of this refactoring, we should also ensure that we have only one queue per Producer instance.

        Attachments

        1. kafka-253_v2.patch
          175 kB
          Jun Rao
        2. kafka-253_v3.patch
          175 kB
          Jun Rao
        3. kafka-253_v4.patch
          177 kB
          Jun Rao
        4. kafka-253_v5.patch
          176 kB
          Jun Rao
        5. kafka-253.patch
          166 kB
          Jun Rao

          Issue Links

            Activity

              People

              • Assignee:
                junrao Jun Rao
                Reporter:
                nehanarkhede Neha Narkhede
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 168h
                  168h
                  Remaining:
                  Remaining Estimate - 168h
                  168h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified