[NIFI-5601] Add fragment.* attributes to GenerateTableFetch

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.9.0
• Component/s: Extensions
• Assignee: Matt Burgess
• Reporter: Matt Burgess
• Created: 15/Sep/18 17:35
• Updated: 04/Feb/19 18:55
• Resolved: 04/Feb/19 18:55

Description

GenerateTableFetch will generate SQL statements in multiple flow files corresponding to "pages" of data for a single execution of the processor. QueryDatabaseTable does something similar, although it also does the fetching of the rows and adds fragment.* attributes to the outgoing flow files, in case a merge or other correlation needs to be done later.

GenerateTableFetch (although it doesn't produce source data per se) is usually used in conjunction with ExecuteSQL to fetch the actual pages of rows, and sometimes these would need to be merged/correlated. To that end, this Jira proposes to add the same fragment.* attributes to the outgoing GTF flow files for use downstream.

Note that if the SQL statements are distributed among a cluster (for parallel execution by ExecuteSQL), they will not be able to be merged, as different nodes may get different fragments and often no single node will get all the fragments.

Issue Links

• links to: GitHub Pull Request #3074