[NIFI-2865] Address issues of PublishKafka blocking when having trouble communicating with Kafka broker and improve performance

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: Extensions
• Assignee: Mark Payne
• Reporter: Mark Payne
• Created: 04/Oct/16 19:22
• Updated: 13/Oct/16 18:54
• Resolved: 06/Oct/16 20:12

Description

When NiFi is unable to communicate properly with the Kafka broker, we see the NiFi threads occasionally block. This should be resolvable by calling the wakeup() method of the client. Additionally, if Kafka takes too long to respond, we should be able to route the FlowFile to failure and move on.

PublishKafka has a nice feature that allows a demarcated stream to be sent as separate messages, so that a large number of messages can be sent as a single FlowFile. However, in the case of individual messages per FlowFile, the performance could be improved by batching together multiple FlowFiles per session

Issue Links

• is related to: NIFI-2463 Improve error message for PublishKafka
• links to: GitHub Pull Request #1097