[NIFI-13301] Create ExtensionRegistryClient for External Extension Registry Interaction

• Type: New Feature
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: Extensions, NiFi Registry
• Environment: Linux: Ubuntu 20.04 or 22.04
• Assignee: James Guzman (Medel)
• Reporter: James Guzman (Medel)
• Created: 27/May/24 06:50
• Updated: 04/Jun/24 17:52

Description

Objective: Improve/enable sharing/reuse of at least two features of Apache NiFi, so the community can have an easier time contributing their flows and/or custom processors into a NiFi Marketplace:

• Pre-Designed Flows: (Approach 1) stored in a NiFi Registry or accessible location. (Approach 2) NiFi Registry can also substituted by NiFi's git registry client having the location of the NiFi Marketplace.  Thus, we just have a git location and NiFi uses that to find/store flows.
• Components/Processors built in Java/Python that anyone is free to use
• End to End Full Stack Applications Powered By NiFi or MiNiFi CPP? The frontend could be various ones from PyQT, ReactJS, H2O.ai Wave, 3D Slicer, OHIF Viewer, etc: (Maybe this one can be subscription based where users get access if they invest in the monthly or yearly subscription?)

Potential Solution: Apache NiFi builds an extension point for interacting with Extension Registries, similar to how it currently has FlowRegistryClient with implementations for NiFi Registry and now GitHub, there would be ExtensionRegistryClient with implementations for stuff like Nexus, NiFi Registry, and any other vendor ones, for example maybe Datavolo provides one, but basically in apache we don't want to build another application, we already have NiFi Registry. (Briefly Discussed with bbende )

• I will start by looking into FlowRegistryClient and then base ExtensionRegistryClient toward that approach. If I am understanding correctly, we would contribute an ExtensionRegistryClient feature into Apache NiFi that enables NiFi to integrate with other vendors (Datavolo, H2Oai, etc) group of processors, data flow templates and so on. I see where you are coming from with for Apache the goal being not to build another application. That application could be by the another and NiFi's ExtensionRegistryClient hooks up to their RegistryServer. (Heres the path I will take toward that goal).

 

Project Ownership: There will be a clear line that the NiFi Marketplace is not owned by the Apache NiFi PMC, rather it will be managed and owned by a vendor in close alignment with NiFi, which we will announce at a later time.

 

Motivation (NiFi): Some people in the community have shown interest in having a NiFi Processor Marketplace where they can contribute their category of processors. In particular, I saw some people interested in contributing their NiFi python processors. This NiFi Processor marketplace could be also applied toward the community interested in contributing custom NiFi java processors.

• Extra MiNiFi C+{}: We could even add a section for MiNiFi C+ custom processors. There is a part of the MiNiFi build process that brings in NiFi through building the JNI extension. So, that is a way to integrate NiFi Registry there too.

I have briefly talked with bbende and joewitt asking them questions on ways to bring custom python processors into production. One of the routes was through leveraging NiFi Registry in connection with Apache NiFi to streamline integration. For my use case, I would leverage the NiFi Processor Marketplace to start contributing my AI/DL "Medical Imaging" Pipeline of custom NiFi Python Processors where I focused on my master thesis AI/DL for stroke diagnosis.

High Level Details of NiFi Processor Marketplace App: For a full stack NiFi Processor Marketplace App, there are frameworks we can leverage for the UI from ReactJS (this mainly is used for frontend), H2O.ai Wave (this can be used for full stack, but probably would mainly be used for the frontend portion) and other options. in the case that we want to make something more custom to display each category of processors. This full stack application could interact wth the NiFi Registry, which is the central location for storage and management of shared resources across one or more instances of NiFi and/or MiNiFi (C++). What are some hosting options for Apache projects?

Issue Links

• relates to: NIFI-13077 On-demand Extension Provider