[NIFI-8002] Create new Elasticsearch Scroll/Paginated Query processor using REST API

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.12.1
• Fix Version/s: 1.15.0
• Assignee: Chris Sampson
• Reporter: Chris Sampson
• Created: 14/Nov/20 00:44
• Updated: 17/Oct/22 16:24
• Resolved: 05/Oct/21 04:34

Description

The existing ScrollElasticsearch processor does not fully work with newer versions of Elasticsearch (e.g. 7.x).

A new version of this processor should be created using the REST API approach, allowing for a query to be passed as input FlowFile content. Appropriate parameters should be set as processor properties (e.g. index, scroll size, scroll time) and allow for additional query string parameters to be set as dynamic properties (see NIFI-8001).

The processor should support both _scroll and _pit/search_after. While scroll may be backward compatible, _pit was introduced in ES 7 (an earlier version of search_before exists for <7, but suggest not trying to support that as part of this ticket).

It may be possible to simply extend the existing REST API's Query processor to page through results with the selected method, rather than create a new processor.

Issue Links

• fixes: NIFI-6458 ElasticSearchClientServiceImpl does not support Elasticserch 7 due to changes in search responses
• is related to: NIFI-8001 Elasticsearch REST processors should allow dynamic properties as query string parameters
• links to: GitHub Pull Request #5193