[BEAM-5865] Auto sharding of streaming sinks in FlinkRunner

• Type: Improvement
• Status: Triage Needed
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.14.0
• Component/s: runner-flink
• Assignee: Jozef Vilcek
• Reporter: Maximilian Michels
• Created: 26/Oct/18 09:25
• Updated: 13/Apr/23 10:55
• Resolved: 15/May/19 08:58

Description

The Flink Runner should do auto-sharding of streaming sinks, similar to BEAM-1438. That way, the user doesn't have to set shards manually which introduces additional shuffling and might cause skew in the distribution of data.

As per discussion: https://lists.apache.org/thread.html/7b92145dd9ae68da1866f1047445479f51d31f103d6407316bb4114c@%3Cuser.beam.apache.org%3E

Issue Links

• is related to: BEAM-1438 The default behavior for the Write transform doesn't work well with the Dataflow streaming runner
• relates to: BEAM-2873 Detect number of shards for file sink in Flink Streaming Runner
• links to: GitHub Pull Request #8499