Kafka
  1. Kafka
  2. KAFKA-332

Mirroring should use multiple producers; add producer retries to DefaultEventHandler

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.1
    • Component/s: core
    • Labels:
      None

      Description

      I'm clubbing these two together as these are both important for mirroring.

      (1) Multiple producers:

      Shallow iteration (KAFKA-315) helps improve mirroring throughput when
      messages are compressed. With shallow iteration, the mirror-maker's consumer
      does not do deep iteration over compressed messages. However, when its
      embedded producer sends these messages to the target cluster's brokers, the
      receiving broker does deep iteration to validate the messages before
      appending to the log.

      In the current (pre- KAFKA-48) request handling mechanism, one producer
      effectively translates to one server-side thread for handling produce
      requests, so there is still a bottleneck due to decompression (due to
      message validation) on the target broker.

      One way to work around this is to use broker.list with multiple brokers
      specified per broker. E.g.,
      broker.list=0:localhost:9191,1:localhost:9191,2:localhost:9191,... which
      effectively emulates multiple server-side threads. It would be better to
      just add a num.producers option to the mirror-maker and instantiate that
      many producers.

      (2) Retries:

      If the mirror-maker uses broker.list and one of the brokers is bounced for
      any reason, messages can get lost. Message loss can be reduced/avoided if
      the brokers are behind a VIP and if retries are supported. This option will
      not work for the zk-based producer because the decision of which broker to
      send to has already been made, so retries would go to the same (potentially
      still down) broker. (With KAFKA-253 it would work for zk-based producers as
      well, but that is only in 0.8).

      1. KAFKA-332-v1.patch
        11 kB
        Joel Koshy
      2. KAFKA-332-v2.patch
        11 kB
        Joel Koshy
      3. KAFKA-332-v3.patch
        12 kB
        Joel Koshy

        Activity

        Jun Rao made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.7.1 [ 12319140 ]
        Resolution Fixed [ 1 ]
        Joel Koshy made changes -
        Attachment KAFKA-332-v3.patch [ 12524101 ]
        Joel Koshy made changes -
        Attachment KAFKA-332-v2.patch [ 12523057 ]
        Joel Koshy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Joel Koshy made changes -
        Field Original Value New Value
        Attachment KAFKA-332-v1.patch [ 12522625 ]
        Joel Koshy created issue -

          People

          • Assignee:
            Joel Koshy
            Reporter:
            Joel Koshy
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development