[NIFI-13301] Create ExtensionRegistryClient for External Extension Registry Interaction - ASF JIRA

Attach files

Attach Screenshot

Add vote

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: New Feature
Status: Open
Priority: Minor
Resolution: Unresolved
Affects Version/s: None
Fix Version/s: None
Component/s: Extensions, NiFi Registry
Labels:
None
Environment:
Linux: Ubuntu 20.04 or 22.04

Description

Objective: Improve/enable sharing/reuse of at least two features of Apache NiFi, so the community can have an easier time contributing their flows and/or custom processors into a NiFi Marketplace:

Pre-Designed Flows: (Approach 1) stored in a NiFi Registry or accessible location. (Approach 2) NiFi Registry can also substituted by NiFi's git registry client having the location of the NiFi Marketplace. Thus, we just have a git location and NiFi uses that to find/store flows.
Components/Processors built in Java/Python that anyone is free to use
End to End Full Stack Applications Powered By NiFi or MiNiFi CPP? The frontend could be various ones from PyQT, ReactJS, H2O.ai Wave, 3D Slicer, OHIF Viewer, etc: (Maybe this one can be subscription based where users get access if they invest in the monthly or yearly subscription?)

Potential Solution: Apache NiFi builds an extension point for interacting with Extension Registries, similar to how it currently has FlowRegistryClient with implementations for NiFi Registry and now GitHub, there would be ExtensionRegistryClient with implementations for stuff like Nexus, NiFi Registry, and any other vendor ones, for example maybe Datavolo provides one, but basically in apache we don't want to build another application, we already have NiFi Registry. (Briefly Discussed with Bryan Bende )

I will start by looking into FlowRegistryClient and then base ExtensionRegistryClient toward that approach. If I am understanding correctly, we would contribute an ExtensionRegistryClient feature into Apache NiFi that enables NiFi to integrate with other vendors (Datavolo, H2Oai, etc) group of processors, data flow templates and so on. I see where you are coming from with for Apache the goal being not to build another application. That application could be by the another and NiFi's ExtensionRegistryClient hooks up to their RegistryServer. (Heres the path I will take toward that goal).

Project Ownership: There will be a clear line that the NiFi Marketplace is not owned by the Apache NiFi PMC, rather it will be managed and owned by a vendor in close alignment with NiFi, which we will announce at a later time.

Motivation (NiFi): Some people in the community have shown interest in having a NiFi Processor Marketplace where they can contribute their category of processors. In particular, I saw some people interested in contributing their NiFi python processors. This NiFi Processor marketplace could be also applied toward the community interested in contributing custom NiFi java processors.

Extra MiNiFi C+{}: We could even add a section for MiNiFi C+ custom processors. There is a part of the MiNiFi build process that brings in NiFi through building the JNI extension. So, that is a way to integrate NiFi Registry there too.

I have briefly talked with Bryan Bende and Joe Witt asking them questions on ways to bring custom python processors into production. One of the routes was through leveraging NiFi Registry in connection with Apache NiFi to streamline integration. For my use case, I would leverage the NiFi Processor Marketplace to start contributing my AI/DL "Medical Imaging" Pipeline of custom NiFi Python Processors where I focused on my master thesis AI/DL for stroke diagnosis.

High Level Details of NiFi Processor Marketplace App: For a full stack NiFi Processor Marketplace App, there are frameworks we can leverage for the UI from ReactJS (this mainly is used for frontend), H2O.ai Wave (this can be used for full stack, but probably would mainly be used for the frontend portion) and other options. in the case that we want to make something more custom to display each category of processors. This full stack application could interact wth the NiFi Registry, which is the central location for storage and management of shared resources across one or more instances of NiFi and/or MiNiFi (C++). What are some hosting options for Apache projects?