[NIFI-3356] Provide a newly refactored provenance repository

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Core Framework
• Assignee: Mark Payne
• Reporter: Mark Payne
• Created: 16/Jan/17 19:15
• Updated: 11/Aug/17 17:53
• Resolved: 22/Feb/17 17:45

Description

The Persistent Provenance Repository has been redesigned a few different times over several years. The original design for the repository was to provide storage of events and sequential iteration over those events via a Reporting Task. After that, we added the ability to compress the data so that it could be held longer. We then introduced the notion of indexing and searching via Lucene. We've since made several more modifications to try to boost performance.

At this point, however, the repository is still the bottleneck for many flows that handle large volumes of small FlowFiles. We need a new implementation that is based around the current goals for the repository and that can provide better throughput.

Issue Links

• contains: NIFI-811 Average Task Duration metric should be more granular
• contains: NIFI-1439 Consistency in pop-up Stats page
• is duplicated by: NIFI-388 Develop new Provenance Repository that provides near-real-time access to events
• relates to: NIFI-3594 Implement encrypted provenance repository
• relates to: NIFI-3712 NiFi shutdown is delayed using WriteAheadProvenanceRepository
• relates to: NIFI-3388 Provide encrypted repository implementations
• links to: GitHub Pull Request #1493