[NIFI-7642] KafkaConsumers: Improve batching into fewer FlowFiles under high latency conditions

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.7.1, 1.9.0, 1.10.0, 1.9.1, 1.9.2, 1.11.0, 1.11.1, 1.11.2, 1.11.4
• Component/s: Extensions
• Labels: kafka
• Assignee: Unassigned
• Reporter: Alex Goos
• Created: 15/Jul/20 10:16
• Updated: 11/May/21 00:04

Description

NIFI-3962 introduced two hard-coded magic numbers into KafkaConsumers. :

• maxRecords is capped at 1000, regardless of the property setting
• the poll timeout is fixed at 10ms

Under high throughput & high latency conditions, this leads to too small small files.

Issue Links

• is related to: NIFI-3962 Update ConsumeKafkaRecord_0_10 to better batch together messages into fewer flowfiles
• links to: GitHub Pull Request #4410