[CAMEL-15228] camel-kafka - Missing end of polling signal in Processors

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.5.0
• Component/s: camel-kafka
• Estimated Complexity: Moderate
• Assignee: Ramu
• Reporter: Jörg
• Created: 22/Jun/20 12:42
• Updated: 01/Jul/20 19:45
• Resolved: 01/Jul/20 19:45

Description

For batch requirements it is currently not possible to get a signal for the last result within a poll request in processors (over all partitions and topics).

The problem with this is, that a Processor cannot know when the last record of a poll received and it is time for commits to all of the affected partitions, because you cannot be sure that there is always a content in every partition and therefore you have no info in Processer, if any further message arrives.

Solution could be to add a property to the manual commit handler in the KafkaConsumer.KafkaFetchRecords.doRun

 

exchange.getIn().setHeader(KafkaConstants.MANUAL_COMMIT, manual);

//adding new exchange property
exchange.getIn().setHeader(KafkaConstants.LAST_RECORD_OF_CURRENT_POLL_REQUEST, last_record_of_last_partition);

LAST_RECORD_IN_CURRENT_POLL_REQUEST should be a boolean to signal end of the loop over all partitions

 

Issue Links

• links to: GitHub Pull Request #3960