Details
-
New Feature
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
We often struggle with scaling custom processors in large flows. We've implemented a recent design pattern where we write custom python code packaged into a docker image, fronted with a web-service, and deployed into a container environment. In NiFi we use InvokeHTTP to call out to the web-service. If the load becomes too great, we can easily spin-up more than one external web-service without sacrificing the NiFi's overall thruput and thread contention. In this scenario, we're using NiFi to manage the overall dataflow, routes, ETL services, etc, but not so much smarts around auto-scaling processing, etc.
This makes it really easy to receive a dataflow analytic/enrichment from an external team, wrap it in a docker image, and deploy it into an auto-scaling container environment. This also lets us avoid conversations about how performant the code is and if it's meeting a minimum throughput requirement - typically XXGB/5minutes.
The request for 2.x would be to make interacting with container deployed services a more seamless, streamlined process, possibly through a normalized API or as a "Remote Container Environment" that can bind to a service with a health endpoint and verify it's alive and well, etc.