Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-986

Expose audit information on flow design changes to reporting tasks

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4.0
    • Component/s: None
    • Labels:
      None

      Description

      It would be helpful for reporting tasks to have access to the audit information about flow design changes in order to report this information to external systems.

      The first use case for this would be reporting flow design changes to Atlas in order for Atlas to capture the flow design metadata.

      1. NIFI-986.patch
        164 kB
        Bryan Bende
      2. NIFI-986-2.patch
        166 kB
        Bryan Bende
      3. NIFI-986-3.patch
        173 kB
        Bryan Bende
      4. NIFI-986-4.patch
        172 kB
        Bryan Bende
      5. NIFI-986-5.patch
        179 kB
        Bryan Bende
      6. nifi-flowchanges-bundle.tar.gz
        7 kB
        Bryan Bende

        Activity

        Hide
        bende Bryan Bende added a comment -

        This patch adds a new method to the EventAccess interface:

        List<Action> getFlowChanges(int firstActionId, final int maxActions);
        

        In addition, interfaces were created for the classes in nifi-user-actions and placed in nifi-api.

        Show
        bende Bryan Bende added a comment - This patch adds a new method to the EventAccess interface: List<Action> getFlowChanges( int firstActionId, final int maxActions); In addition, interfaces were created for the classes in nifi-user-actions and placed in nifi-api.
        Hide
        bende Bryan Bende added a comment -

        Fixing a bug in MockEventAccess and rebasing.

        Show
        bende Bryan Bende added a comment - Fixing a bug in MockEventAccess and rebasing.
        Hide
        bende Bryan Bende added a comment -

        Matt Gilman patch should be ready for review

        Show
        bende Bryan Bende added a comment - Matt Gilman patch should be ready for review
        Hide
        mcgilman Matt Gilman added a comment -

        -1

        I don't believe the current patch would provide access to events when clustered. Currently, actions are stored in the NCM and are bundled in the response from the nodes. ClusteredEventAccess.getFlowChanges would need to be implemented.

        It makes sense to also store the actions locally on the nodes when clustered. We should bundle the events to be stored at the NCM when the instance is a connected node. However, we should always save the events in the local action DB.

        Mark Payne Is there a reason why we don't want to support Primary Node Only scheduling strategy for reporting tasks? I think there may be a use case for it here since the actions may be batched at the NCM. I see that previously I was only adding support for Timer and CRON. I would add support conditionally based on the Reporting Task Availability.

        Show
        mcgilman Matt Gilman added a comment - -1 I don't believe the current patch would provide access to events when clustered. Currently, actions are stored in the NCM and are bundled in the response from the nodes. ClusteredEventAccess.getFlowChanges would need to be implemented. It makes sense to also store the actions locally on the nodes when clustered. We should bundle the events to be stored at the NCM when the instance is a connected node. However, we should always save the events in the local action DB. Mark Payne Is there a reason why we don't want to support Primary Node Only scheduling strategy for reporting tasks? I think there may be a use case for it here since the actions may be batched at the NCM. I see that previously I was only adding support for Timer and CRON. I would add support conditionally based on the Reporting Task Availability.
        Hide
        bende Bryan Bende added a comment -

        Thanks for reviewing Matt. New patch addresses your feedback about clustered mode and always saving locally.

        Also attaching a tar of a reporting task bundle which you can build and deploy test the changes. It just prints all the flow changes on each execution.

        Show
        bende Bryan Bende added a comment - Thanks for reviewing Matt. New patch addresses your feedback about clustered mode and always saving locally. Also attaching a tar of a reporting task bundle which you can build and deploy test the changes. It just prints all the flow changes on each execution.
        Hide
        mcgilman Matt Gilman added a comment -

        +1 This looks good Bryan! Patch applies cleanly, contrib check passes, and verified functionality in both standalone and clustered mode.

        Once Mark Payne chimes in about Primary Node only for Reporting Tasks scheduling strategy we can open up a new ticket for that effort. Thanks.

        Show
        mcgilman Matt Gilman added a comment - +1 This looks good Bryan! Patch applies cleanly, contrib check passes, and verified functionality in both standalone and clustered mode. Once Mark Payne chimes in about Primary Node only for Reporting Tasks scheduling strategy we can open up a new ticket for that effort. Thanks.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 5cc2b04b918a0a3c76c0f1d71c7a416d0dce3f39 in nifi's branch refs/heads/master from Bryan Bende
        [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=5cc2b04 ]

        NIFI-986 Refactoring of action classes from nifi-user-actions to have interfaces in nifi-api, and adding getFlowChanges to EventAccess

        • Fixing empty java docs and adding sort by id asc to the history query
        • Changing userDn to userIdentity in Action and FlowChangeAction
        • Modifying NiFiAuditor to always save events locally, and implementing getFlowChanges for ClusteredEventAccess
        Show
        jira-bot ASF subversion and git services added a comment - Commit 5cc2b04b918a0a3c76c0f1d71c7a416d0dce3f39 in nifi's branch refs/heads/master from Bryan Bende [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=5cc2b04 ] NIFI-986 Refactoring of action classes from nifi-user-actions to have interfaces in nifi-api, and adding getFlowChanges to EventAccess Fixing empty java docs and adding sort by id asc to the history query Changing userDn to userIdentity in Action and FlowChangeAction Modifying NiFiAuditor to always save events locally, and implementing getFlowChanges for ClusteredEventAccess
        Hide
        bende Bryan Bende added a comment -

        Pushed to master

        Show
        bende Bryan Bende added a comment - Pushed to master
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 5cc2b04b918a0a3c76c0f1d71c7a416d0dce3f39 in nifi's branch refs/heads/NIFI-655 from Bryan Bende
        [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=5cc2b04 ]

        NIFI-986 Refactoring of action classes from nifi-user-actions to have interfaces in nifi-api, and adding getFlowChanges to EventAccess

        • Fixing empty java docs and adding sort by id asc to the history query
        • Changing userDn to userIdentity in Action and FlowChangeAction
        • Modifying NiFiAuditor to always save events locally, and implementing getFlowChanges for ClusteredEventAccess
        Show
        jira-bot ASF subversion and git services added a comment - Commit 5cc2b04b918a0a3c76c0f1d71c7a416d0dce3f39 in nifi's branch refs/heads/ NIFI-655 from Bryan Bende [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=5cc2b04 ] NIFI-986 Refactoring of action classes from nifi-user-actions to have interfaces in nifi-api, and adding getFlowChanges to EventAccess Fixing empty java docs and adding sort by id asc to the history query Changing userDn to userIdentity in Action and FlowChangeAction Modifying NiFiAuditor to always save events locally, and implementing getFlowChanges for ClusteredEventAccess

          People

          • Assignee:
            bende Bryan Bende
            Reporter:
            bende Bryan Bende
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development