Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-7773

JoltTransformJSON mixing up flow files order

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • 1.11.4
    • None
    • Core Framework
    • Ubuntu 16 x64, 3 nodes cluster
    • Important

    Description

      Using JoltTransformJSON to transform (flatten/rename fields) kafka JSON messages before making an update on Postgres DB

      In some rare cases messages (received from same topic partition) are "mixing" up an order within a same node
      Therefore updates are invoked in wrong order, affecting on data consistency and correctness

      Provenance logs for some found cases:
      Partition and offset for each message provided in screen (with blue)

      Flow (mentioned "flow pipeline" marked with yellow):

      Flow code has been attached

      Attachments

        1. image-2020-08-28-13-06-03-269.png
          217 kB
          Dmytro Kulyk
        2. image-2020-08-28-13-06-20-964.png
          245 kB
          Dmytro Kulyk
        3. image-2020-08-28-13-07-39-001.png
          312 kB
          Dmytro Kulyk
        4. Sportsbook (2).json
          76 kB
          Dmytro Kulyk

        Activity

          People

            Unassigned Unassigned
            kotyara Dmytro Kulyk
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: