[NIFI-7745] Add a SampleRecord processor

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.13.0
• Component/s: Extensions
• Assignee: Matt Burgess
• Reporter: Matt Burgess
• Created: 15/Aug/20 17:04
• Updated: 02/Sep/20 16:26
• Resolved: 02/Sep/20 16:25

Description

Sampling records in a flowfile can be a helpful way to test with "real" data, especially for source systems that contain large datasets. It may not be possible on the source system to sample the data or test NiFi flows on smaller datasets from the source system(s). Sampling in NiFi may be currently possible (such as QueryRecord with row numbers), but is likely done in-memory (in the QueryRecord case) or in a simplistic fashion.

This Jira proposes a SampleRecord processor that should offer (at the least) the following sampling options:

Interval Sampling (every Nth record)
Probabilistic Sampling (each record has a probability P of being chosen)
Reservoir Sampling (A sample of size K with each record having equal probability of being chosen)

Issue Links

• links to: GitHub Pull Request #4482