Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-7773

JoltTransformJSON mixing up flow files order

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.11.4
    • Fix Version/s: None
    • Component/s: Core Framework
    • Labels:
    • Environment:
      Ubuntu 16 x64, 3 nodes cluster
    • Flags:
      Important

      Description

      Using JoltTransformJSON to transform (flatten/rename fields) kafka JSON messages before making an update on Postgres DB

      In some rare cases messages (received from same topic partition) are "mixing" up an order within a same node
      Therefore updates are invoked in wrong order, affecting on data consistency and correctness

      Provenance logs for some found cases:
      Partition and offset for each message provided in screen (with blue)

      Flow (mentioned "flow pipeline" marked with yellow):

      Flow code has been attached

        Attachments

        1. image-2020-08-28-13-06-03-269.png
          217 kB
          Dmytro Kulyk
        2. image-2020-08-28-13-06-20-964.png
          245 kB
          Dmytro Kulyk
        3. image-2020-08-28-13-07-39-001.png
          312 kB
          Dmytro Kulyk
        4. Sportsbook (2).json
          76 kB
          Dmytro Kulyk

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              kotyara Dmytro Kulyk
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: