1. Kafka
  2. KAFKA-249

Separate out Kafka mirroring into a stand-alone app


    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.1
    • Component/s: core
    • Labels:


      I would like to discuss on this jira, the feasibility/benefits of separating
      out Kafka's mirroring feature from the broker into a stand-alone app, as it
      currently has a couple of limitations and issues.

      For example, we recently had to deal with Kafka mirrors that were in fact
      idle due to the fact that mirror threads were not created at start-up due to
      a rebalancing exception, but the Kafka broker itself did not shutdown. This
      has since been fixed, but is indicative of (avoidable) problems in embedding
      non-broker specific features in the broker.

      Logically, it seems to make sense to separate it out to achieve better
      division of labor. Furthermore, enhancements to mirroring may be less
      clunky to implement and use with a stand-alone app. For example to support
      custom partitioning on the target cluster, or to mirror from multiple
      clusters we would probably need to be able to pass in multiple embedded
      consumer/embedded producer configs, which would be less ugly if the
      mirroring process were a stand-alone app. Also, if we break it out, it
      would be convenient to use as a "consumption engine" for the console
      consumer which will make it easier to add on features such as wildcards in
      topic consumption, since it contains a ZooKeeper topic discovery component.

      Any suggestions and/or objections to this?

      1. KAFKA-249.v5.patch
        192 kB
        Joel Koshy
      2. KAFKA-249.v4-v5.incremental.patch
        13 kB
        Joel Koshy
      3. KAFKA-249.v4.patch
        191 kB
        Joel Koshy
      4. KAFKA-249.v3-v4.incremental.patch
        67 kB
        Joel Koshy
      5. KAFKA-249.v3.patch
        189 kB
        Joel Koshy
      6. KAFKA-249.v2.patch
        113 kB
        Joel Koshy
      7. KAFKA-249.v1.patch
        73 kB
        Joel Koshy


        Joel Koshy created issue -
        Joel Koshy made changes -
        Field Original Value New Value
        Attachment KAFKA-249.v1.patch [ 12515790 ]
        Joel Koshy made changes -
        Attachment KAFKA-249.v2.patch [ 12520044 ]
        Joel Koshy made changes -
        Attachment KAFKA-249.v3.patch [ 12521084 ]
        Joel Koshy made changes -
        Attachment KAFKA-249.v4.patch [ 12521421 ]
        Joel Koshy made changes -
        Attachment KAFKA-249.v3-v4.incremental.patch [ 12521422 ]
        Joel Koshy made changes -
        Attachment KAFKA-249.v5.patch [ 12521780 ]
        Joel Koshy made changes -
        Attachment KAFKA-249.v4-v5.incremental.patch [ 12521781 ]
        Jun Rao made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]


          • Assignee:
            Joel Koshy
            Joel Koshy
          • Votes:
            1 Vote for this issue
            1 Start watching this issue


            • Created: