Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-3196

Provide MiNiFi FlowStatus Insights processor that integrates MiNiFi Flow Status Report with Azure Application Insights

    Details

    • Type: New Feature
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.1.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      As a user of both NiFi and MiNiFi there are many times I would like to have an operational dashboard for MiNiFi. Although, there is a discussion and desire for Centralized management there are many times where as a user I just want to be able to use the operational tools I already have. In many cases those may be products such as: Nagios, Zenoos, Zabix, Graphite, Grafana and in cloud environments perhaps Azure Application Insights and AWS Cloudwatch.

      This JIRA and the related P/R provides a processor that ingests the MiNiFi Flow Status report and sends it to Azure Application Insights as custom metrics. This then allows a user to set alerts in Azure Application Insights, thus providing the ability to monitor MiNiFi agents. As such, this has a dependency on the following MiNiFi JIRA: Change FlowStatusReport to be JSON

        Issue Links

          Activity

          Hide
          aldrin Aldrin Piri added a comment -

          Sounds like a good concept. Curious if it makes sense to deconstruct this a bit more but do not have great familiarity with the Azure App Insights (AAI). Could see a series of processors that receive the data, then transform (transform JSON) and then a PutAAI or similar. That last bit of functionality seems like it might get some nice reuse for a lot of other uses.

          Show
          aldrin Aldrin Piri added a comment - Sounds like a good concept. Curious if it makes sense to deconstruct this a bit more but do not have great familiarity with the Azure App Insights (AAI). Could see a series of processors that receive the data, then transform (transform JSON) and then a PutAAI or similar. That last bit of functionality seems like it might get some nice reuse for a lot of other uses.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user apsaltis opened a pull request:

          https://github.com/apache/nifi/pull/1331

          NIFI-3196 Provide MiNiFi FlowStatus Insights processor that integrates MiNiFi Flow Status Report with Azure Application Insights

          As a user of both NiFi and MiNiFi there are many times I would like to have an operational dashboard for MiNiFi. Although, there is a discussion and desire for Centralized management there are many times where as a user I just want to be able to use the operational tools I already have. In many cases those may be products such as: Nagios, Zenoos, Zabix, Graphite, Grafana and in cloud environments perhaps Azure Application Insights and AWS Cloudwatch.

          This P/R provides a processor that ingests the MiNiFi Flow Status report and sends it to Azure Application Insights as custom metrics. This then allows a user to set alerts in Azure Application Insights, thus providing the ability to monitor MiNiFi agents. As such, this has a dependency on the following MiNiFi JIRA: Change FlowStatusReport to be JSON (https://github.com/apache/nifi-minifi/pull/65(url))

          An example flow of how to set this up for the MiNiFi side is:

          ![image](https://cloud.githubusercontent.com/assets/1100275/21208171/61891462-c23a-11e6-9075-e7657653f40f.png)

          And the NiFi Side looks like:
          ![image](https://cloud.githubusercontent.com/assets/1100275/21208215/bc114cec-c23a-11e6-8393-ee2ec50e61e7.png)

          Then once this is all set in Azure Application Insights I can build a dashboard that may look like this:
          ![image](https://cloud.githubusercontent.com/assets/1100275/21208329/67804e02-c23b-11e6-8e7d-933c56f33d3b.png)

          And explore all the various metrics, that I may want to build a dashboard from:
          ![image](https://cloud.githubusercontent.com/assets/1100275/21208347/913e5b76-c23b-11e6-89b6-b8895210ffff.png)

          I can then also go and set alerts
          ![image](https://cloud.githubusercontent.com/assets/1100275/21208458/128b653e-c23c-11e6-8737-d8665c280325.png)

          In the end this type of integration can also be done with other enterprise tools.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/apsaltis/nifi NIFI-3196

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/nifi/pull/1331.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #1331


          commit 9d3fa636a4f0feb9c7055488c5f686dd3f237e36
          Author: Andrew Psaltis <psaltis.andrew@gmail.com>
          Date: 2016-12-14T16:40:27Z

          NIFI-3196 Provide MiNiFi FlowStatus Insights processor that integrates MiNiFi Flow Status Report with Azure Application Insights


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user apsaltis opened a pull request: https://github.com/apache/nifi/pull/1331 NIFI-3196 Provide MiNiFi FlowStatus Insights processor that integrates MiNiFi Flow Status Report with Azure Application Insights As a user of both NiFi and MiNiFi there are many times I would like to have an operational dashboard for MiNiFi. Although, there is a discussion and desire for Centralized management there are many times where as a user I just want to be able to use the operational tools I already have. In many cases those may be products such as: Nagios, Zenoos, Zabix, Graphite, Grafana and in cloud environments perhaps Azure Application Insights and AWS Cloudwatch. This P/R provides a processor that ingests the MiNiFi Flow Status report and sends it to Azure Application Insights as custom metrics. This then allows a user to set alerts in Azure Application Insights, thus providing the ability to monitor MiNiFi agents. As such, this has a dependency on the following MiNiFi JIRA: Change FlowStatusReport to be JSON ( https://github.com/apache/nifi-minifi/pull/65 (url)) An example flow of how to set this up for the MiNiFi side is: ! [image] ( https://cloud.githubusercontent.com/assets/1100275/21208171/61891462-c23a-11e6-9075-e7657653f40f.png ) And the NiFi Side looks like: ! [image] ( https://cloud.githubusercontent.com/assets/1100275/21208215/bc114cec-c23a-11e6-8393-ee2ec50e61e7.png ) Then once this is all set in Azure Application Insights I can build a dashboard that may look like this: ! [image] ( https://cloud.githubusercontent.com/assets/1100275/21208329/67804e02-c23b-11e6-8e7d-933c56f33d3b.png ) And explore all the various metrics, that I may want to build a dashboard from: ! [image] ( https://cloud.githubusercontent.com/assets/1100275/21208347/913e5b76-c23b-11e6-89b6-b8895210ffff.png ) I can then also go and set alerts ! [image] ( https://cloud.githubusercontent.com/assets/1100275/21208458/128b653e-c23c-11e6-8737-d8665c280325.png ) In the end this type of integration can also be done with other enterprise tools. You can merge this pull request into a Git repository by running: $ git pull https://github.com/apsaltis/nifi NIFI-3196 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1331.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1331 commit 9d3fa636a4f0feb9c7055488c5f686dd3f237e36 Author: Andrew Psaltis <psaltis.andrew@gmail.com> Date: 2016-12-14T16:40:27Z NIFI-3196 Provide MiNiFi FlowStatus Insights processor that integrates MiNiFi Flow Status Report with Azure Application Insights
          Hide
          apsaltis Andrew Psaltis added a comment -

          It would certainly be possible to put other processors in-front of this to do transformation of the data. The only thing this knows about the input is that it is a MiNiFi FlowStatus report serialized as JSON and from that it deserializes it and sends it along as metrics to AAI.

          Show
          apsaltis Andrew Psaltis added a comment - It would certainly be possible to put other processors in-front of this to do transformation of the data. The only thing this knows about the input is that it is a MiNiFi FlowStatus report serialized as JSON and from that it deserializes it and sends it along as metrics to AAI.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user trixpan commented on the issue:

          https://github.com/apache/nifi/pull/1331

          @apsaltis - loved the idea behind this contribution - I'm also very much in favor of fitting MiNiFi with existing systems - and perhaps that's the reason for my first question:

          Is there a reason why to bundle the Azure Insights as a bundle tied to this NAR? Looking at the metrics collected it seems fairly useful to NiFi as well and suspect people running NiFi in the cloud would rave about being able to send reporting data directly into insights instead of having to rely solely on Ambari and other existing reporting services.

          It may be just me but I suspect AzureApplicationInsights if empowered by Expression Language enabled Dynamic Properties would be a killing processor...

          What do you think?

          Show
          githubbot ASF GitHub Bot added a comment - Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1331 @apsaltis - loved the idea behind this contribution - I'm also very much in favor of fitting MiNiFi with existing systems - and perhaps that's the reason for my first question: Is there a reason why to bundle the Azure Insights as a bundle tied to this NAR? Looking at the metrics collected it seems fairly useful to NiFi as well and suspect people running NiFi in the cloud would rave about being able to send reporting data directly into insights instead of having to rely solely on Ambari and other existing reporting services. It may be just me but I suspect AzureApplicationInsights if empowered by Expression Language enabled Dynamic Properties would be a killing processor... What do you think?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user apsaltis commented on the issue:

          https://github.com/apache/nifi/pull/1331

          @trixpan I agree that having something much more generic would be great. I am totally open to closing this PR, rebasing my branch and taking this processor in that direction. Honestly, after having time to think about it It being tied to MiNiFi and talking with others I totally agree.

          Show
          githubbot ASF GitHub Bot added a comment - Github user apsaltis commented on the issue: https://github.com/apache/nifi/pull/1331 @trixpan I agree that having something much more generic would be great. I am totally open to closing this PR, rebasing my branch and taking this processor in that direction. Honestly, after having time to think about it It being tied to MiNiFi and talking with others I totally agree.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user trixpan commented on the issue:

          https://github.com/apache/nifi/pull/1331

          @apsaltis - glad comments were helpful.

          New PR or not, the work is truly great and I can confess that with the exception of Azure components I will very likely be using it the outcome of the new contribution as soon as it gets merged.

          Show
          githubbot ASF GitHub Bot added a comment - Github user trixpan commented on the issue: https://github.com/apache/nifi/pull/1331 @apsaltis - glad comments were helpful. New PR or not, the work is truly great and I can confess that with the exception of Azure components I will very likely be using it the outcome of the new contribution as soon as it gets merged.

            People

            • Assignee:
              apsaltis Andrew Psaltis
              Reporter:
              apsaltis Andrew Psaltis
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development