[FLINK-6316] Kinesis connector docs should mention that FlinkKinesisConsumer does not provide strong ordering guarantees on resharding

• Type: Improvement
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Component/s: Connectors / Kinesis, Documentation
• Labels: auto-deprioritized-major auto-deprioritized-minor
• Assignee: Unassigned
• Reporter: Tzu-Li (Gordon) Tai
• Created: 18/Apr/17 00:52
• Updated: 20/Dec/21 22:38

Description

Since the FlinkKinesisConsumer depends only on local information to determine whether or not a new shard due to resharding should be subscribed by a subtask, there is no coordination wrt parent-child shard relationship across subtasks.

Therefore, FlinkKinesisConsumer does not provide any strong processing ordering guarantees.

Take for example, if initially the assignment is:
Subtask #1 --> Shard A
Subtask #2 --> Shard B

Assume A & B is merged to create shard C, and subtask #1 locally determines that it should be assigned shard C.

Since Flink generally does not provide coordinating facilities between subtask instances, there is no means of coordinating that shard C is consumed only after shard B is also fully consumed.