Major Description
I would like to discuss on this jira, the feasibility/benefits of separating
out Kafka's mirroring feature from the broker into a stand-alone app, as it
currently has a couple of limitations and issues.
For example, we recently had to deal with Kafka mirrors that were in fact
idle due to the fact that mirror threads were not created at start-up due to
a rebalancing exception, but the Kafka broker itself did not shutdown. This
has since been fixed, but is indicative of (avoidable) problems in embedding
non-broker specific features in the broker.
Logically, it seems to make sense to separate it out to achieve better
division of labor. Furthermore, enhancements to mirroring may be less
clunky to implement and use with a stand-alone app. For example to support
custom partitioning on the target cluster, or to mirror from multiple
clusters we would probably need to be able to pass in multiple embedded
consumer/embedded producer configs, which would be less ugly if the
mirroring process were a stand-alone app. Also, if we break it out, it
would be convenient to use as a "consumption engine" for the console
consumer which will make it easier to add on features such as wildcards in
topic consumption, since it contains a ZooKeeper topic discovery component.
Any suggestions and/or objections to this?