Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
1.11.4, 1.12.1
-
None
-
None
Description
PUT Elasticsearch should support the new Elasticsearch Data Streams (new in Elasticsearch 7.9).
NIFI-7474 will allow these processors to submit create operations via the _bulk API (which is a large part of the requirement).
However, Data Streams require an @timestamp field to be provided in each document, but this field name is illegal in Avro schemas due to the leading @. The Record-based processors should therefore allow for the injection of this field into the JSON being sent to Elasticsearch - this could be based upon an existing field within the FlowFile and be identified by a property on the processor (e.g. like the _id field can be specified using Record Path).
Optionally, the processor allow for the field used as the @timestamp field to be removed from the data being sent to Elasticsearch (i.e. rename the existing field or duplicate it depending upon property settings). Such field transformation should also take the timestamp format settings into account(e.g. if a Long epoch millisecond value is to be converted to a formatted date/time String).
