Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-13829

MonitorActivity processor generating false inactivity files

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.27.0, 2.0.0-M4
    • 1.28.0, 2.0.0
    • Extensions
    • None

    Description

      MonitorActivity generates false inactivity reports (with big ratio), if the FlowFiles are coming less frequent, than the 2/3 of the configured threshold. This happens only in cluster monitoring scope, where only primary node reports, and only non-primary nodes have FlowFiles to process.

      This is because, the processor synchronizes the common state only 3 times within the threshold: 1T/3, 2T/3, 3T/3. If the FlowFile hits the non-primary node between the second (2T/3) and the third sync point (3T/3), then the state from the non-primary node will only be shared with the primary node, when the last sync point comes. And as sync points are relativeley calculated by each node, there is a race condition in terms of which happens first: the primary node reads the common state, or the non-primary node writes the new state. In case of the former, primary node will generate a false indication.

      [PRIMARY] - SY0 ---- SY1 ----- SY2 ---- SY3 ---- SY4 ----
                           /         /        /      / ? /
      [C STATE] ------X---:---------:--------:-------:X-:------
                     /        /        /             /
      [SLAVE N] -- FF1 ---- SY1 ---- SY2 -- FF2 -- SY3 --------
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            takraj Rajmund Takacs
            takraj Rajmund Takacs
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment