[NIFI-8689] Site-to-Site client is constantly flushing the socket's OutputStream

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.14.0
• Component/s: Core Framework
• Assignee: Mark Payne
• Reporter: Mark Payne
• Created: 11/Jun/21 17:18
• Updated: 14/Jun/21 07:32
• Resolved: 11/Jun/21 20:15

Description

When a RemoteProcessGroup is sending data to another NiFi instance, the protocol should establish a transaction and then send a sequence of FlowFiles following a pattern along the lines of:

<FlowFile Follows><FlowFile Attributes><FlowFile Content>
<FlowFile Follows><FlowFile Attributes><FlowFile Content>
<FlowFile Follows><FlowFile Attributes><FlowFile Content>
<Finished Transaction>

However, currently, the protocol is flushing the Socket's output buffer each that that it indicates that a FlowFile follows, and again after each FlowFile. So it's more like:

<FlowFile Follows>*Flush Buffer*
<FlowFile Attributes><FlowFile Content>*Flush Buffer*
<FlowFile Follows>*Flush Buffer*
<FlowFile Attributes><FlowFile Content>*Flush Buffer*
<FlowFile Follows>*Flush Buffer*
<FlowFile Attributes><FlowFile Content>*Flush Buffer*

As a result, when sending a large number of smaller FlowFiles, we end up constantly flushing data to the socket, which results in dramatically worse performance.

Issue Links

• links to: GitHub Pull Request #5150