[KAFKA-15597] Enable Connect DropHeaders SMT to drop headers on a wildcard/regexp-basis

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: connect
• Labels: connect-transformation
• Assignee: Unassigned
• Reporter: Roman Schmitz
• Created: 12/Oct/23 10:22
• Updated: 16/Oct/23 07:39

Description

In many use cases you might not only want to drop a few specific Kafka headers but a set of headers whose names can also dynamically change (e.g. when used with some end-to-end-encryption libraries, tracing etc.). To prevent those headers following a special pattern (which may not comply with downstream system format) to be further forwarded/processed downstream (e.g.header forwarding in Http Sinks), I suggest to add regexp matching to the apply method instead of a set-based contains. Link to the relevant code:
https://github.com/apache/kafka/blob/7b5d640cc656443a078bda096d01910b3edfdb37/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/DropHeaders.java#L54 

One common example is forwarding of encryption-headers that contain byte arrays to a HTTP server via HTTP Sinks which will not be accepted by the HTTP server.

Example implementation done - link to PR: https://github.com/apache/kafka/pull/14536

Issue Links

• links to: GitHub Pull Request #14536
• mentioned in: Page Loading...