[FLINK-30725] FLIP-281: Sink Supports Speculative Execution For Batch Job

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 1.17.0
• Component/s: Runtime / Coordination

Release Note

Flink now allows sinks which implement decorative interface SupportsConcurrentExecutionAttempts to do speculative execution. If a sink(including Sink V2, SinkFunction and OutputFormat) can work properly when multiple attempts are executing at the same time, it can implement SupportsConcurrentExecutionAttempts.
Some built in sinks are enabled to do speculative execution, including DiscardingSink, PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat and HiveTableSink.

More details can be found at https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/speculative_execution.

• Assignee: Biao Liu
• Reporter: Biao Liu
• Due: 31/Jan/23
• Created: 18/Jan/23 04:00
• Updated: 10/Feb/23 12:46
• Resolved: 06/Feb/23 09:59

Description

This is the umbrella issue of FLIP-281. More details can be found in https://cwiki.apache.org/confluence/display/FLINK/FLIP-281+Sink+Supports+Speculative+Execution+For+Batch+Job.

Issue Links

• causes: FLINK-31005 Release Testing: Verify FLIP-281 Supports speculative execution of sinks

Sub-Tasks

• Make SinkV2 support speculative execution for batch jobs: Closed
• Make OutputFormat support speculative execution for batch jobs: Closed
• Make SinkFunction support speculative execution for batch jobs: Closed
• Enable speculative execution for some of the typical built-in sinks: Closed
• Add document for sink supports speculative execution: Closed