[NIFI-11240] Introduce Python API for building Processors

• Type: Epic
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0-M1
• Component/s: Core Framework, Documentation & Website, Extensions
• Epic Name: Introduce Python API for building Processors
• Epic Status: To Do
• Assignee: Mark Payne
• Reporter: Mark Payne
• Created: 02/Mar/23 18:08
• Updated: 15/Nov/23 08:32
• Resolved: 03/Nov/23 21:20

Description

The scripting processors are very common for data transformation in NiFi. In particular, the Jython based scripts are quite heavily used. However, Jython is run on the JVM and does not support CPython libraries. As a result, it's syntax compatible but doesn't make use of the wealth of Python libraries. And the wealth of Python libraries are what make Python popular to begin with.

Additionally, use of many script-based processors hurts the UX. They are cumbersome to configure, with script files and/or script bodies. They result in a dataflow that's difficult to understand because instead of nicely named processors like CompressContent the type and default name are "ExecuteScript." They're also difficult to share.

I have been playing with Py4J for introduce a true Python-based API for developing Processors. This will introduce new APIs, new framework changes, and documentation. And this will likely take a while to stabilize. However, the sooner that we are able to land it into the hands of users, the better. Therefore, I pose that we introduce it in multiple milestones. We can create sub-tickets for different milestones, but in general it should follow:

Milestone 1: Initial implementation. Provides the capability and an API for building processors. Includes sample code and some documentation. Includes tests to ensure proper operation. Should not be used in production. API will not be stable and may change frequently. Performance may be subpar. Get into the hands of developers to begin exploring and providing feedback / submitting PRs.

Milestone 2: Bug fixes. API refinement. Improve performance.

Milestone 3: Additional bug fixes and API refinement. API should become more stable.

Milestone 4: Additional bug fixes. API becomes stable. Documentation is clear and sufficient. Recommend production use.

 

 

Sub-Tasks

• Introduce API for building Processors in Python: Resolved
• Implement Dependent Properties in Python Processors: Resolved
• Speed up process of adding a Python Processor: Resolved
• Support usage of requirements.txt: Resolved