Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-8689

Site-to-Site client is constantly flushing the socket's OutputStream

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.14.0
    • Component/s: Core Framework
    • Labels:
      None

      Description

      When a RemoteProcessGroup is sending data to another NiFi instance, the protocol should establish a transaction and then send a sequence of FlowFiles following a pattern along the lines of:

      <FlowFile Follows><FlowFile Attributes><FlowFile Content>
      <FlowFile Follows><FlowFile Attributes><FlowFile Content>
      <FlowFile Follows><FlowFile Attributes><FlowFile Content>
      <Finished Transaction>

      However, currently, the protocol is flushing the Socket's output buffer each that that it indicates that a FlowFile follows, and again after each FlowFile. So it's more like:

      <FlowFile Follows>*Flush Buffer*
      <FlowFile Attributes><FlowFile Content>*Flush Buffer*
      <FlowFile Follows>*Flush Buffer*
      <FlowFile Attributes><FlowFile Content>*Flush Buffer*
      <FlowFile Follows>*Flush Buffer*
      <FlowFile Attributes><FlowFile Content>*Flush Buffer*

      As a result, when sending a large number of smaller FlowFiles, we end up constantly flushing data to the socket, which results in dramatically worse performance.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                markap14 Mark Payne
                Reporter:
                markap14 Mark Payne
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m