Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-5922

Stateless NiFi, an alternative runtime for NiFi flows

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.10.0
    • Component/s: None
    • Labels:
      None

      Description

      NiFi-Fn is a library for running NiFi flows as stateless functions. It provides similar delivery guarantees as NiFi without the need for on-disk repositories by waiting to confirm receipt ofincoming data until it has been written to the destination. This is similar to Storm’s acking mechanism and Spark’s interface for committing Kafka offsets, except that in NiFi-Fn, this is completely handled by the framework while still supporting all NiFi processors and controller services natively without change.This results in the ability to run NiFi flows as ephemeral, stateless functions and should be able to rival MirrorMaker, Distcp, and Scoop for performance,efficiency, and scalability while leveraging the vast library of NiFi processors and the NiFi UI for building custom flows.

      By leveraging container engines (e.g.YARN, Kubernetes), long-running NiFi-Fn flows can be deployed that take full advantage of the platform’s scale and multi-tenancy features. By leveraging Function as a Service engines (FaaS) (e.g. AWS Lambda, Apache OpenWhisk), NiFi-Fn flows can be attached to event sources (or just cron) for event-driven data movement where flows only run when triggered and pricing is measured at the 100ms granularity. By combining the two, large-scale batch processing could also be performed.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              samhjelmfelt Sam Hjelmfelt
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 4h 40m
                4h 40m