Stateless NiFi, an alternative runtime for NiFi flows

NiFi-Fn is a library for running NiFi flows as stateless functions. It provides similar delivery guarantees as NiFi without the need for on-disk repositories by waiting to confirm receipt ofincoming data until it has been written to the destination. This is similar to Storm’s acking mechanism and Spark’s interface for committing Kafka offsets, except that in NiFi-Fn, this is completely handled by the framework while still supporting all NiFi processors and controller services natively without change.This results in the ability to run NiFi flows as ephemeral, stateless functions and should be able to rival MirrorMaker, Distcp, and Scoop for performance,efficiency, and scalability while leveraging the vast library of NiFi processors and the NiFi UI for building custom flows.

By leveraging container engines (e.g.YARN, Kubernetes), long-running NiFi-Fn flows can be deployed that take full advantage of the platform’s scale and multi-tenancy features. By leveraging Function as a Service engines (FaaS) (e.g. AWS Lambda, Apache OpenWhisk), NiFi-Fn flows can be attached to event sources (or just cron) for event-driven data movement where flows only run when triggered and pricing is measured at the 100ms granularity. By combining the two, large-scale batch processing could also be performed.