Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5487

Proper at-least-once support for ElasticsearchSink

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0
    • Component/s: Streaming Connectors
    • Labels:
      None

      Description

      Discussion in ML: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fault-tolerance-guarantees-of-Elasticsearch-sink-in-flink-elasticsearch2-td10982.html

      Currently, the Elasticsearch Sink actually doesn't offer any guarantees for message delivery.

      For proper support of at-least-once, the sink will need to participate in Flink's checkpointing: when snapshotting is triggered at the ElasticsearchSink, we need to synchronize on the pending ES requests by flushing the internal bulk processor. For temporary ES failures (see FLINK-5122) that may happen on the flush, we should retry them before returning from snapshotting and acking the checkpoint. If there are non-temporary ES failures on the flush, the current snapshot should fail.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tzulitai Tzu-Li (Gordon) Tai
                Reporter:
                tzulitai Tzu-Li (Gordon) Tai
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: