Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-5429

Improve failure monitoring

    XMLWordPrintableJSON

    Details

      Description

      A user in the ASF HipChat room and I discussed the following scenario:

      With N processors in a flow, I want to send a notification on any failure event to a remote monitoring system. For this example, let the remote monitoring system be Elasticsearch. Currently, I need N unique PutElasticsearch processors to uniquely identify the processor which has the failure.

      I created some diagrams showing the existing setup and desired setup (one with direct to reporting; one with an ExecuteScript processor for metadata enrichment (i.e. if the component UUID is available, use ES to resolve the human-friendly name and processor group nesting to be persisted)).

      There are concerns about exposing component ID to flowfiles and how this can be misused, but there needs to be a better solution for how to monitor flow failures without 1:1 processors for monitoring.

      I have also attached the conversation.

      cc: Mark Payne Joe Witt

        Attachments

        1. asf_hc_failure_reporting.txt
          9 kB
          Andy LoPresto
        2. asf_hc_failure_reporting.003.jpeg
          182 kB
          Andy LoPresto
        3. asf_hc_failure_reporting.002.jpeg
          154 kB
          Andy LoPresto
        4. asf_hc_failure_reporting.001.jpeg
          223 kB
          Andy LoPresto

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              alopresto Andy LoPresto
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: