Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-6271

ExecuteSQL incoming flowfile attributes not copied into output flowfiles when Output Batch Size is set

    XMLWordPrintableJSON

    Details

      Description

      When using the executeSQL and executeSQLRecord processors, we can use input flowfiles with a certain number of attributes. If we don't set the Output Batch Size, all these attributes are copied to the output flowfile. However, if we set it, only the flowfiles from the first batch will have the attributes copied to. The flowfiles in the following batches will only have the default attributes. 

      Root cause

      In the source code of the method onTrigger in the class AbstractExecuteSQL, we have the following piece of code that is supposed to create an output flowfile and copy the original attributes into it: 

      FlowFile resultSetFF;
      if (fileToProcess == null) {
        resultSetFF = session.create();
      } else {
        resultSetFF = session.create(fileToProcess);
        resultSetFF = session.putAllAttributes(resultSetFF, fileToProcess.getAttributes());
      }
      

      However the fix for the issue NIFI-6040 introduced this snippet way below in the same method: 

       

      // If we've reached the batch size, send out the flow files
      if (outputBatchSize > 0 && resultSetFlowFiles.size() >= outputBatchSize) {
        session.transfer(resultSetFlowFiles, REL_SUCCESS);
        // Need to remove the original input file if it exists
        if (fileToProcess != null) {
          session.remove(fileToProcess);
          fileToProcess = null;
        }
        session.commit();
        resultSetFlowFiles.clear();
      }
      

      As you can see, it sets the variable fileToProcess to null, preventing the flowfiles in the next batch to copy its attributes

       

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hondawei HondaWei
                Reporter:
                bosion Arnaud Rivero
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 0.5h Original Estimate - 0.5h
                  0.5h
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h
                  2h