[NIFI-8021] Allow ConsumeKafka processors to use static partition mapping

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.13.0
• Component/s: Extensions
• Assignee: Mark Payne
• Reporter: Mark Payne
• Created: 17/Nov/20 22:27
• Updated: 24/Mar/21 14:02
• Resolved: 06/Jan/21 18:32

Description

With the ConsumeKafka processors, we use the KafkaConsumer.subscribe() method, which is generally what is desirable. However, for a case where strict FIFO order per partition is critical, as may be the case with CDC use cases among others, we should allow users to pin a particular partition to a particular node. This way, even if the node is restarted or stops pulling data for a while, its partitions are not re-assigned, so there's no risk of messages being delivered out of order.

Issue Links

• is fixed by: NIFI-8357 ConsumeKafka(Record)_2_0, ConsumeKafka(Record)_2_6 do not reconnect if using statically assigned partitions
• links to: GitHub Pull Request #4672
• links to: GitHub Pull Request #4744