Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-9878

DistributedCacheMap Handshake failure, processor hang indefinitely.

    XMLWordPrintableJSON

Details

    Description

      When a DistributedCacheMapClient attempts to connect to a DistributedCacheMapServer, but the handshake response is never received by the client, the PutDistributedCacheMap processor with hang indefinitely. The handshake never times out.

      A situation like this can be caused if a proxy allows for the TCP connection to be established between client and server but fails to deliver handshake data to/from DistributedCacheMapServer (for example an unstable Istio service mesh between the two). Could also happen if a client was accidentally misconfigured to point to wrong TCP server point (one that wasn't hosting a DistributedCacheMapServer.

      Steps to recreate:

      1) Set up a PutDistributedCacheMap processor with a DistributedMapCacheClientService

      2) Configure DistributedMapCacheClientService to point to a non DistributedCacheMapServer tcp server (nc -lk 127.0.0.1 4457). This simulates a situation where the socket connection can be made but there is no handshake response from the server (for example, server is in bad state and unable to respond, a proxy is misbehaving, etc).

      3) use generateFlowFile to trigger PutDistributedCacheMap  processor.

      4) processor will hang with no failure or success. Processor will have to be force terminated.

      Hang occurs at :
      CacheClientRequestHandler.java:92: handshakeHandler.waitHandshakeComplete();
       
      Currently, the "connection timeout" parameter is only used to timeout the establishment of the TCP socket connection, not the full application layer connection.
      Suggestion:

      Handshake should have a timeout too to be robust to handle a network outage where the TCP connection is able to be created, but the handshake data can't be exchanged. The processor hanging prevents any way to handle this error in a dataflow.

       

      Attachments

        Issue Links

          Activity

            People

              Jon Shoemaker Jon Shoemaker
              arich Aaron Rich
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1.5h
                  1.5h