Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-4376

Expired requests stay in StansdardRootGroupPort request queue and block new request to be offered

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.1
    • Fix Version/s: 1.4.0
    • Component/s: Core Framework
    • Labels:
      None

      Description

      If the remote NiFi instance slows down for some reason, and S2S requests have been timed out continuously, those timed out requests will remain in the internal request queue at StandardRootGroupPort.

      Once number of queued requests becomes 1,000, newly received request can not be offered to the queue, and RequestExpiredException is thrown. A S2S client receives following error in this case:

      2017-09-11 16:32:38,137 ERROR [Timer-Driven Process Thread-4] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=output,target=http://localhost:8080/nifi] failed to communicate with http://localhost:8080/nifi due to java.io.IOException: Unexpected response code: 500 errCode:Abort errMessage:<html>
      <head>
      <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
      <title>Error 500 Server Error</title>
      </head>
      <body><h2>HTTP ERROR 500</h2>
      <p>Problem accessing /nifi-api/data-transfer/output-ports/6fc7fa6d-015e-1000-27cb-d118b1b0fe13/transactions/73da9bcf-87ec-4594-8e45-a41965c1ed82/flow-files. Reason:
      <pre>    Server Error</pre></p><h3>Caused by:</h3><pre>java.io.IOException: Failed to process the request.
              at org.apache.nifi.web.api.DataTransferResource$1.write(DataTransferResource.java:676)
              at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71)
      ...
      <h3>Caused by:</h3><pre>org.apache.nifi.remote.exception.RequestExpiredException
              at org.apache.nifi.remote.StandardRootGroupPort.transferFlowFiles(StandardRootGroupPort.java:562)
              at org.apache.nifi.web.api.DataTransferResource$1.write(DataTransferResource.java:668)
      

      Once a NiFi instance owning the root group port enters this state, it will not be able to recover automatically as its client keeps retrying too often. And the returned error message is not clear to understand what is happening.

      The remote NiFi instance should remove already expired requests from its internal queue, so that newly received requests are processed correctly, and return 'read timeout' error response if it can not be processed and expired.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ijokarumawak Koji Kawamura
                Reporter:
                ijokarumawak Koji Kawamura
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: