[NIFI-6070] Elasticsearch HTTP processors should use Low-level Java REST client

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.9.0
• Assignee: Unassigned
• Reporter: Aaron Leon
• Created: 21/Feb/19 22:32
• Updated: 23/Feb/19 00:06
• Resolved: 22/Feb/19 18:18

Description

The Elasticsearch HTTP processors should use Java REST client... Either high-level (which would tie the processor to specific versions of Elasticsearch in the same way the transport client processors are, e.g. PutElasticsearch5) or the low-level, which from my understanding does not depend on a specific version of ES. The reasoning is that the current Elasticsearch HTTP processors simply use OkHTTP to hit the REST API, but this does not include critical features to a production setup such as:

• load balancing across all available nodes
• failover in case of node failures and upon specific response codes
• failed connection penalization (whether a failed node is retried depends on how many consecutive times it failed; the more failed attempts the longer the client will wait before trying that same node again)
• persistent connections
• trace logging of requests and responses
• optional automatic discovery of cluster nodes

(source: https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-low.html)

The current PutElasticsearchHTTP processor also sends timed out requests to 'failure' queue as opposed to 'retry' which is not the correct behavior.

Issue Links

• relates to: NIFI-5248 Create PutElasticsearchRecord processor using the Elasticsearch client service