[FLINK-5487] Proper at-least-once support for ElasticsearchSink

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.3.0
• Component/s: Connectors / Common
• Assignee: Tzu-Li (Gordon) Tai
• Reporter: Tzu-Li (Gordon) Tai
• Created: 13/Jan/17 21:11
• Updated: 24/Feb/17 15:40
• Resolved: 24/Feb/17 15:40

Description

Discussion in ML: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fault-tolerance-guarantees-of-Elasticsearch-sink-in-flink-elasticsearch2-td10982.html

Currently, the Elasticsearch Sink actually doesn't offer any guarantees for message delivery.

For proper support of at-least-once, the sink will need to participate in Flink's checkpointing: when snapshotting is triggered at the ElasticsearchSink, we need to synchronize on the pending ES requests by flushing the internal bulk processor. For temporary ES failures (see FLINK-5122) that may happen on the flush, we should retry them before returning from snapshotting and acking the checkpoint. If there are non-temporary ES failures on the flush, the current snapshot should fail.

Issue Links

• is blocked by: FLINK-4988 Elasticsearch 5.x support
• is related to: FLINK-5122 Elasticsearch Sink loses documents when cluster has high load
• links to: GitHub Pull Request #3358