[NIFI-7953] Allow ConsumeKafka_2_* / ConsumeKafkaRecord_2_* to separate Records based on Message Key

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.13.0
• Component/s: Extensions
• Assignee: Mark Payne
• Reporter: Mark Payne
• Created: 27/Oct/20 15:39
• Updated: 27/Oct/20 23:04
• Resolved: 27/Oct/20 23:04

Description

The Kafka consumer processors ignore the Message Key when using a demarcator or when using the Record variant, because adding a FlowFile attribute for the message key doesn't make sense, since the key is likely different per record.

However, a fairly common use case is to use the Key only for partitioning and many Kafka Messages may exist for the same key. For example, this is commonly used for CDC type of use cases where the key may just be a table name.

For such cases, it would be helpful to allow the consumer processors to avoid adding two messages to the same FlowFile if their keys are different. I.e., all records in a given FlowFile will have the same key. The key can then be added as a FlowFile attribute.

Issue Links

• links to: GitHub Pull Request #4625