Kafka
  1. Kafka
  2. KAFKA-253

Refactor the async producer to have only one queue instead of one queue per broker in a Kafka cluster

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      Today, the async producer is associated with a particular broker instance, just like the SyncProducer. The Producer maintains a producer pool of sync/async producers, one per broker. Since the producer pool creates one async producer per broker, we have multiple producer queues for one Producer instance.

      With replication, a topic partition will be logical. This requires refactoring the AsyncProducer to be broker agnostic. As a side effect of this refactoring, we should also ensure that we have only one queue per Producer instance.

      1. kafka-253_v2.patch
        175 kB
        Jun Rao
      2. kafka-253_v3.patch
        175 kB
        Jun Rao
      3. kafka-253_v4.patch
        177 kB
        Jun Rao
      4. kafka-253_v5.patch
        176 kB
        Jun Rao
      5. kafka-253.patch
        166 kB
        Jun Rao

        Issue Links

          Activity

          Neha Narkhede created issue -
          Neha Narkhede made changes -
          Field Original Value New Value
          Link This issue blocks KAFKA-239 [ KAFKA-239 ]
          Neha Narkhede made changes -
          Link This issue incorporates KAFKA-233 [ KAFKA-233 ]
          Jun Rao made changes -
          Attachment kafka-253.patch [ 12512775 ]
          Jun Rao made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jun Rao made changes -
          Attachment kafka-253_v2.patch [ 12513029 ]
          Jun Rao made changes -
          Attachment kafka-253_v3.patch [ 12513092 ]
          Jun Rao made changes -
          Attachment kafka-253_v4.patch [ 12513163 ]
          Jun Rao made changes -
          Attachment kafka-253_v5.patch [ 12513461 ]
          Jun Rao made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.8 [ 12317244 ]
          Resolution Fixed [ 1 ]

            People

            • Assignee:
              Jun Rao
              Reporter:
              Neha Narkhede
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 168h
                168h
                Remaining:
                Remaining Estimate - 168h
                168h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development