Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-249

Separate out Kafka mirroring into a stand-alone app


    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.1
    • Component/s: core
    • Labels:


      I would like to discuss on this jira, the feasibility/benefits of separating
      out Kafka's mirroring feature from the broker into a stand-alone app, as it
      currently has a couple of limitations and issues.

      For example, we recently had to deal with Kafka mirrors that were in fact
      idle due to the fact that mirror threads were not created at start-up due to
      a rebalancing exception, but the Kafka broker itself did not shutdown. This
      has since been fixed, but is indicative of (avoidable) problems in embedding
      non-broker specific features in the broker.

      Logically, it seems to make sense to separate it out to achieve better
      division of labor. Furthermore, enhancements to mirroring may be less
      clunky to implement and use with a stand-alone app. For example to support
      custom partitioning on the target cluster, or to mirror from multiple
      clusters we would probably need to be able to pass in multiple embedded
      consumer/embedded producer configs, which would be less ugly if the
      mirroring process were a stand-alone app. Also, if we break it out, it
      would be convenient to use as a "consumption engine" for the console
      consumer which will make it easier to add on features such as wildcards in
      topic consumption, since it contains a ZooKeeper topic discovery component.

      Any suggestions and/or objections to this?


        1. KAFKA-249.v1.patch
          73 kB
          Joel Koshy
        2. KAFKA-249.v2.patch
          113 kB
          Joel Koshy
        3. KAFKA-249.v3.patch
          189 kB
          Joel Koshy
        4. KAFKA-249.v3-v4.incremental.patch
          67 kB
          Joel Koshy
        5. KAFKA-249.v4.patch
          191 kB
          Joel Koshy
        6. KAFKA-249.v4-v5.incremental.patch
          13 kB
          Joel Koshy
        7. KAFKA-249.v5.patch
          192 kB
          Joel Koshy



            • Assignee:
              jjkoshy Joel Koshy
              jjkoshy Joel Koshy
            • Votes:
              1 Vote for this issue
              1 Start watching this issue


              • Created: