[NIFI-7971] List-Fetch-Delete pattern

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.12.1
• Component/s: Extensions
• Assignee: Unassigned
• Reporter: Dennis Jaheruddin
• Created: 01/Nov/20 11:20
• Updated: 01/Nov/20 11:21

Description

Challenge

There are several processors for getting data. Originally we had GetHDFS, GetSFTP and these are now being phased out for List Fetch processors. 

I think the List-Fetch pattern is a significant improvement, but what I see now is that people are looking for a List-Fetch-Delete pattern. 

Example usecase:

Gather data, process and write in destination, delete

There are several ways to achieve this (native but inefficient ones, like 'do a second Fetch with remove source file and terminate success', or efficient but scripted ones, like 'use the executestreamprocessor to send the delete command').

As far as I understand, this would not be needed when using Stateless Nifi, but that is not relevant for many usecases.

Proposal

However, I propose we find a solution that allows people to easily achieve a List-Fetch-Delete pattern. 

Two lines of thought:

1. Expand Fetch processors with a 'deleteonly' mode, which would not actually copy in the data
2. Create dedicated processors, like 'DeleteSFTP' or 'ExecuteSFTP'

I have written the example for SFTP, but think this would be a natural extension for  most places where we have the List-Fetch pattern currently.