[NIFI-6728] InvokeHttp / PostHttp should yield on HTTP-503 (Service Unavailable)

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.7.1
• Component/s: Core Framework
• Assignee: Unassigned
• Reporter: Andrew Doumaux
• Created: 30/Sep/19 12:50
• Updated: 30/Sep/19 12:50

Description

Problem:

InvokeHttp/PostHttp only penalize a flowfile when a the remote host returns an error code.  It has no way to yield the processor when the remote host indicates it's unavailable.  

 

Desired Behavior:

A configurable property to have the processor yield on server error codes (5xx).  Specifically HTTP-503.

Maybe user exceptions as well (4xx), specifically HTTP-401 & HTTP-403 .

 

Use Case:

I'm using NIFI to load balance data to multiple horizontally scalable ETL processes (Standalone java spring boot apps).  The ETL application returns a HTTP-503 when it is saturated with work and have no available threads to process a new piece of data.  I have a separate PostHttp processor for each ETL application instance/host.  It was done this way since I had the misconception that NIFI would yield that specific processor and thus pause sending data to that ETL application as it worked off it's current load.  

This behavior of penalizing the file and not yielding the processor is especially problematic when attempting to process an excessive backlog of data.