[FLINK-13886] Need to develop flink-connector-elasticsearch7

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.8.0, 1.8.1, 1.9.0
• Component/s: Connectors / ElasticSearch

Release Note

See https://issues.apache.org/jira/browse/FLINK-13025

• Assignee: Unassigned
• Reporter: Leonid Ilyevsky
• Created: 28/Aug/19 14:56
• Updated: 28/Aug/19 19:20
• Resolved: 28/Aug/19 19:20

Description

The flink-connector-elasticsearch6 is not compatible with Elasticsearch 7.x.

If you try to use elasticsearch-rest-high-level-client version 7.x with flink-connector-elasticsearch6 instead of 6.x, you get fatal exception:

java.lang.NoSuchMethodError: org.elasticsearch.client.RestHighLevelClient.ping([Lorg/apache/http/Header;)Z
{{ at org.apache.flink.streaming.connectors.elasticsearch6.Elasticsearch6ApiCallBridge.createClient(Elasticsearch6ApiCallBridge.java:81)}}

......

 

Some changes in Elastic 7.x make it necessary to use compatible client library, specifically the deprecated 'type' element in the document structure. 

With 6.x client library the 'type' element is mandatory, so it has to be in each document, but then the 7.x sever will send back a warning for every single document. Workaround: suppress those warnings in the logging configuration.