[NIFI-10737] ListenBeats Unable to Process Large Batches

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.16.0, 1.17.0, 1.18.0
• Fix Version/s: 1.19.0
• Component/s: Extensions
• Assignee: David Handermann
• Reporter: David Handermann
• Created: 31/Oct/22 13:23
• Updated: 01/Feb/23 03:03
• Resolved: 07/Nov/22 22:11

Description

Refactoring of ListenBeats released in version 1.16.0 introduced processing issues when reading batches of messages larger than 2048 bytes.

The refactored implementation based on Netty introduced a BeatsFrameDecoder extension of the Netty ByteToMessageDecoder, which the Netty framework invokes multiple times with variables contents of the input ByteBuf. The implementation does not handle scenarios where a batch of multiple messages is larger than the buffer size of 2048. As a result of this behavior, the Decoder does not construct a complete message. Without the complete message, the Processor does not send an acknowledgement for the received message, resulting in errors on sending clients, such as the following:

Failed to publish events caused by: read tcp 192.168.1.100:54623->192.168.1.200:5555: i/o timeout

This error occurs on sending client while waiting to receive an acknowledgement packet for the messages transmitted.

As a result of not receiving the acknowledgement, the client attempts to send the same messages multiple times, resulting in duplication and failure to send additional messages.

Issue Links

• fixes: NIFI-3753 ListenBeats: Compressed beats packets may cause: Error decoding Beats frame: Error decompressing frame: invalid distance too far back
• fixes: NIFI-4467 ListenBeats ERROR messages should include remote endpoint (when relevant)
• fixes: NIFI-6738 ListenBeats receives partial messages
• links to: GitHub Pull Request #6608