[FLINK-26281] Test Elasticsearch connector End2End

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.15.0
• Fix Version/s: 1.15.0
• Component/s: Connectors / ElasticSearch
• Labels: pull-request-available release-testing
• Assignee: Konstantin Knauf
• Reporter: Alexander Preuss
• Created: 21/Feb/22 13:23
• Updated: 30/Mar/22 13:50
• Resolved: 30/Mar/22 13:50

Description

Feature introduced in https://issues.apache.org/jira/browse/FLINK-24323

Documentation for datastream api

Documentation for table api

As 1.15 deprecated the SinkFunction-based Elasticsearch connector and introduces the new connector based on the Sink interface we should test it behaves correctly and as the user expects.

 

Some suggestions what to test:

• Test delivery guarantees (none, at-least-once) (exactly-once should not run)
• Write a simple job that is inserting/upserting data into Elasticsearch
• Write a simple job that is inserting/upserting data into Elasticsearch and use a non-default parallelism
• Write a simple job in both datastream api and table api
• Test restarting jobs and scaling up/down
• Test failure of a simple job that is inserting data with exactly-once delivery guarantee by terminating and restarting Elasticsearch
• Test against Elasticsearch 6.X and 7.X with the respective connectors

Issue Links

• links to: GitHub Pull Request #19035
• links to: GitHub Pull Request #19118
• links to: GitHub Pull Request #19200
• links to: GitHub Pull Request #19202
• links to: GitHub Pull Request #19203
• links to: GitHub Pull Request #19277