[NIFI-8113] Persisting status history

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.14.0
• Component/s: Core Framework
• Assignee: Simon Bence
• Reporter: Simon Bence
• Created: 05/Jan/21 08:09
• Updated: 29/Dec/21 16:51
• Resolved: 25/Feb/21 21:23

Description

An alternative solution would be useful to replace/extend current in-memory solution for storing component and node metrics. This is in order to ease heap usage of NiFi which in case of high amount of processors (10-20k) might be significant and also to keep system information after shutting down.

The solution should keep the data even if NiFi is stopped running and make it possible to read it out. It should be optimised for writing data as reading is much less frequent.

 

Proposal:

• Use QuestDB as backend
• Add a new implementation for ComponentStatusRepository
• Making it interchangeable between the existing and upcoming solutions
• Provide some rollback policy in order to avoid ever-growing database

Issue Links

• is duplicated by: NIFI-6866 Create a persistent or stateful stats repository
• is fixed by: NIFI-8289 EmbeddedQuestDbRolloverHandlerTest tests fail when local date and UTC date differ
• links to: GitHub Pull Request #4821
• links to: GitHub Pull Request #4839