[NIFI-697] MergeContent and queue swapping

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.1.0
• Fix Version/s: 0.4.0
• Component/s: Core Framework
• Assignee: Mark Payne
• Reporter: Michael W Moser
• Created: 18/Jun/15 21:56
• Updated: 09/Nov/15 20:43
• Resolved: 09/Nov/15 20:43

Description

MergeContent will not place more files into bins than the nifi.properties nifi.queue.swap.threshold (default 20,000). This is probably worth a documentation note. Perhaps in the Maximum Number of Entries description in BinFiles. It says "The maximum number of files to include in a bundle. If not specified, there is no maximum." The maximum seems to be = nifi.queue.swap.threshold.

Test Case: GenerateFlowFile and put about 81,000 files into a queue to MergeContent. Wait for most of them to be swapped out. Set Minimum Number of Entries to 25000. Set Max Bin Age to 1 min. When you start MergeContent, it will wait the Max Bin Age duration, then it will merge 20,000 files (the nifi.queue.swap.threshold).

Issue Links

• relates to: NIFI-122 FlowFileQueue should block if queue is too full
• relates to: NIFI-516 MergeContent when merging 10,000s or more objects can cause too many open files
• relates to: NIFI-730 Purge Queue from UI