Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-10737

ListenBeats Unable to Process Large Batches

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.16.0, 1.17.0, 1.18.0
    • 1.19.0
    • Extensions
    • None

    Description

      Refactoring of ListenBeats released in version 1.16.0 introduced processing issues when reading batches of messages larger than 2048 bytes.

      The refactored implementation based on Netty introduced a BeatsFrameDecoder extension of the Netty ByteToMessageDecoder, which the Netty framework invokes multiple times with variables contents of the input ByteBuf. The implementation does not handle scenarios where a batch of multiple messages is larger than the buffer size of 2048. As a result of this behavior, the Decoder does not construct a complete message. Without the complete message, the Processor does not send an acknowledgement for the received message, resulting in errors on sending clients, such as the following:

      Failed to publish events caused by: read tcp 192.168.1.100:54623->192.168.1.200:5555: i/o timeout
      

      This error occurs on sending client while waiting to receive an acknowledgement packet for the messages transmitted.

      As a result of not receiving the acknowledgement, the client attempts to send the same messages multiple times, resulting in duplication and failure to send additional messages.

      Attachments

        Issue Links

          Activity

            People

              exceptionfactory David Handermann
              exceptionfactory David Handermann
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m