Major Description
There are many scenarios where there is a need for consumers to subscribe to a message based on pattern eg. Key format or headers.
Typically, this scenario gets solved using external components implementing a filtering logic.
In a deployment with a large number of consumers,This becomes a huger performance (network/IO) overheads if most messages are getting discarded by the consumer based on the filter. If there are 10000's of consumers subscribing to the topic partitions, there is unnecessary IO, which could be avoided broker can apply filter per subscriptions.
I wanted to hear insights from the Kafka community around how they are solving this problem and gauge interest in formally submitting KIP around filtering as part of core capability.
Thanks,
Raj