[NIFI-6215] Cannot swap in Flowfiles

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.9.1, 1.9.2
• Fix Version/s: 1.10.0
• Component/s: Core Framework
• Environment: Windows Server 2012 R2 Standard ; ; java version "1.8.0_181" ; Java(TM) SE Runtime Environment (build 1.8.0_181-b13) ; Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
• Assignee: Mark Payne
• Reporter: Bluemine
• Created: 15/Apr/19 13:00
• Updated: 01/Nov/19 11:18
• Resolved: 15/Apr/19 19:56

Description

Upon creating a lot of FlowFiles simultaneously (though SplitFile or ListSFTP) into a funnel, Nifi reports:

12:15:24 CEST WARNING
Cannot swap in FlowFiles from location C:\PROGRA~1\APACHE~1\NIFI-1~1.2\..\repositories\flowfile_repository\swap\1555323318624-9c6fa68f-38fd-3e05-6f82-1aa48b9995fb-f481b847-ff99-47a9-ab06-6501fb1e9f93.swap because the FlowFile Repository does not know about this Swap Location. This file should be manually removed. This typically occurs when a Swap File is written but the FlowFile Repository is not updated yet to reflect this. This is generally not a cause for concern, but may be indicative of a failure to update the FlowFile Repository.

12:15:30 CEST WARNING
Cannot swap in FlowFiles from location C:\PROGRA~1\APACHE~1\NIFI-1~1.2\..\repositories\flowfile_repository\swap\1555323319139-9c6fa68f-38fd-3e05-6f82-1aa48b9995fb-e7ec580a-dca0-4dec-a65e-9efb468f704d.swap because the FlowFile Repository does not know about this Swap Location. This file should be manually removed. This typically occurs when a Swap File is written but the FlowFile Repository is not updated yet to reflect this. This is generally not a cause for concern, but may be indicative of a failure to update the FlowFile Repository.

[...]

etc.

After handling the first 20,000 files the rest seem to get 'stuck' in the swapfiles and do not get processed. Restarting Nifi erases all files in the queue and clearing the queue in some other fashion is not possible.

Using a funnel seems to trigger the above message, without the funnel the same behaviour occurs but without the message (the swapfiles do get created).

It was not possible to reproduce this behaviour in Unix, so it might be windows-specific, possibly though some weird race-condition.

Attached are a screenshot of what the erroneous state looks like, as well as a template and input file that were able to reproduce the issue.

Issue Links

• duplicates: NIFI-6219 Unprocessable queues with remaining size a multiple of 10000
• links to: GitHub Pull Request #3434