[NIFI-7773] JoltTransformJSON mixing up flow files order

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 1.11.4
• Component/s: Core Framework
• Labels: jolt order
• Environment: Ubuntu 16 x64, 3 nodes cluster
• Flags: Important
• Assignee: Unassigned
• Reporter: Dmytro Kulyk
• Created: 28/Aug/20 10:08
• Updated: 03/Oct/23 06:30

Description

Using JoltTransformJSON to transform (flatten/rename fields) kafka JSON messages before making an update on Postgres DB

In some rare cases messages (received from same topic partition) are "mixing" up an order within a same node
Therefore updates are invoked in wrong order, affecting on data consistency and correctness

Provenance logs for some found cases:
Partition and offset for each message provided in screen (with blue)

Flow (mentioned "flow pipeline" marked with yellow):

Flow code has been attached