[NIFI-6040] ExecuteSQL can't perform batch loading with an input file

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.8.0
• Fix Version/s: 1.10.0, 1.9.1
• Component/s: Core Framework
• Labels: easyfix usability
• Flags: Important
• Assignee: Matt Burgess
• Reporter: Artem Anokhin
• Created: 15/Feb/19 15:21
• Updated: 12/Mar/19 16:23
• Resolved: 06/Mar/19 18:46

Description

Hello!

TC to reproduce:

1) Create GenerateFlowFile with the next Custom Text:

select 1 from dual
union all
select 2 from dual
union all
select 3 from dual

2) Create ExecuteSQL. Set "Output Batch Size" = 1.
3) Start

AR:

ER:

3 flowfiles in the "success" connection.

Root Cause:

if (outputBatchSize > 0 && resultSetFlowFiles.size() >= outputBatchSize) {
    session.transfer(resultSetFlowFiles, REL_SUCCESS);
    session.commit();
    resultSetFlowFiles.clear();
}

Commit method has the next requirement:
Commits the current session ensuring all operations against FlowFiles within this session are atomically persisted. All FlowFiles operated on within this session must be accounted for by transfer or removal or the commit will fail.

At the time of any commit, an input flowfile (fileToProcess) still exists and has not been accounted for by transfer or removal.

Issue Links

• is duplicated by: NIFI-6107 ExecuteSQL processors fail with incoming flow file and Output Batch Size set
• links to: GitHub Pull Request #3355