Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Not A Problem
-
1.13.2
-
None
-
None
-
Java 1.8.0_282, CentOS 7, 8-Node Cluster
Description
Since the upgrade from NiFi 1.11.4 to 1.13.2 we faced an issue with the FetchSFTP & PutSFTP processors. We have a 8-Node NiFi cluster. Pattern is always ListSFTP (tracking timestamp) - FetchSFTP (and delete) and PutSFTP.
If we do a restart of NiFi and NiFi comes back, we sometimes see flowfiles for FetchSFTP (not found) and PutSFTP (already present on disk) which have been processed successfully and have been stored already. So in fact we see flowfiles in a failure queue which have been save to disk with PutSFTP, which should never happen. The files are always small (a few MBs) and the network connectivity is insanely fast. The cluster shutdown is always before the grace period runs out. The attached screeshot shows an example where the FetchSFTP and the PutSFTP failure queue has files. Especially for the FetchSFTP this shouldn't be possible and if I do a restart with the command below, I would expect that within the grace period the processor has been stopped and it can't be processed twice.
/opt/nifi/bin/nifi.sh restart
At the moment we have no clue where the issue comes from and why it happens, so I can't provide an exact scenario to reproduce it. I only know that it sometimes happens after a restart of our 8-node cluster.
