[KAFKA-395] kafka.tools.MirrorMaker black/white list improvement

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Workaround
• Component/s: config
• Assignee: Unassigned
• Reporter: Dave DeMaagd
• Created: 09/Jul/12 19:52
• Updated: 25/Oct/18 13:14
• Resolved: 25/Oct/18 13:13

Description

Current black/white list topics are specified directly on the command line, while functional, this has two drawbacks:
1) Changes become unwieldy if there are a large number of running instances - potentially many instances to restart, which can have implications for data stream lag
2) Maintaining the list itself can become increasingly complex if there are a large number of elements in the list (particularly if they are complex expressions)

Suggest extending the way that black/white lists can be fed to the mirror maker application, in particular, being able to specify the black/white list as a file (or possibly a generic URI). Thinking that this could be accomplished either by adding '-whitelistfile' and 'blacklistfile' command line parameters, or modifying the existing 'blacklist' and '-whitelist' parameters to include a 'is this a valid file?' test and decide how to handle it based on that (if it is a file, read it, if not, use current behavior).

Follow up suggestion would be to have the mirror maker process check for updates to the list file, and on change, validate and reload it, and run from that point with the new list information.

Issue Links

• is superceded by: KAFKA-7500 MirrorMaker 2.0 (KIP-382)