[NIFI-5429] Improve failure monitoring

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.7.0
• Component/s: Core Framework
• Labels: failure monitoring provenance reporting
• Assignee: Unassigned
• Reporter: Andy LoPresto
• Created: 13/Jul/18 18:54
• Updated: 13/Jul/18 18:56

Description

A user in the ASF HipChat room and I discussed the following scenario:

With N processors in a flow, I want to send a notification on any failure event to a remote monitoring system. For this example, let the remote monitoring system be Elasticsearch. Currently, I need N unique PutElasticsearch processors to uniquely identify the processor which has the failure.

I created some diagrams showing the existing setup and desired setup (one with direct to reporting; one with an ExecuteScript processor for metadata enrichment (i.e. if the component UUID is available, use ES to resolve the human-friendly name and processor group nesting to be persisted)).

There are concerns about exposing component ID to flowfiles and how this can be misused, but there needs to be a better solution for how to monitor flow failures without 1:1 processors for monitoring.

I have also attached the conversation.

cc: markap14 joewitt