[NIFI-6970] Add DistributeRecord processor for distribute data by key hash

• Type: New Feature
• Status: Patch Available
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.10.0
• Component/s: Extensions
• External issue URL: https://github.com/apache/nifi/pull/3984
• Assignee: Unassigned
• Reporter: Ilya Kovalev
• Created: 24/Dec/19 17:43
• Updated: 09/Jun/20 16:45

Description

Necessary to add Processor for distribute data over user specified relationships by distribution key/keys. Data is distributed across relationships in the amount proportional to the relationship weight. For example, if there are two relationships and the first has a weight of 9 while the second has a weight of 10, the first will be sent 9 / 19 parts of the rows, and the second will be sent 10 / 19.

The row will be sent to the relationship that corresponds to the half-interval of the remainders from 'prev_weight' to 'prev_weights + weight', where 'prev_weights' is the total weight of the relationships with the smallest number, and 'weight' is the weight of this relationship." For example, if there are two relationships, and the first has a weight of 9 while the second has a weight of 10, the row will be sent to the first relationship for the remainders from the range [0, 9), and to the second for the remainders from the range [9, 19).

 

It will help for loading data to distributed databases like clickhouse https://clickhouse.tech/docs/en/

Issue Links

• links to: GitHub Pull Request #3984