[FLINK-6349] Enforce per-subtask record ordering on resharding for FlinkKinesisConsumer

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Connectors / Kinesis
• Labels: auto-deprioritized-major auto-deprioritized-minor
• Assignee: Unassigned
• Reporter: Tzu-Li (Gordon) Tai
• Created: 21/Apr/17 00:07
• Updated: 24/Nov/23 11:44

Description

As described in FLINK-6316, currently the Kinesis consumer does not provide any ordering guarantees when resharding occurs.

While this cannot be enforced globally (i.e. if a merged / split shard's child shard ends up in a different subtask, we cannot do any coordination for ordering guarantee), we can definitely enforce this locally for each subtask. Simply put, we can still locally enforce ordering by making sure that discovered child shards are consumed only after any of its parent shards that were on the same subtask are fully consumed.

To do this, we would also need to add "parent shard" information to KinesisStreamShard (Flink's representation of Kinesis shards).

This would be directly beneficial for per-shard watermarks (FLINK-5697) to retain per-shard time characteristics after a reshard, and therefore can be seen as a prerequisite.

Issue Links

• blocks: FLINK-5697 Add per-shard watermarks for FlinkKinesisConsumer
• is superceded by: FLINK-32218 Implement support for parent/child shard ordering