[SPARK-23503] continuous execution should sequence committed epochs

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: Structured Streaming
• Assignee: Efim Poberezkin
• Reporter: Jose Torres
• Created: 23/Feb/18 19:13
• Updated: 16/Sep/18 02:01
• Resolved: 18/May/18 23:55

Description

Currently, the EpochCoordinator doesn't enforce a commit order. If a message for epoch n gets lost in the ether, and epoch n + 1 happens to be ready for commit earlier, epoch n + 1 will be committed.

 

This is either incorrect or needlessly confusing, because it's not safe to start from the end offset of epoch n + 1 until epoch n is committed. EpochCoordinator should enforce this sequencing.

 

Note that this is not actually a problem right now, because the commit messages go through the same RPC channel from the same place. But we shouldn't implicitly bake this assumption in.

Issue Links

• links to: [Github] Pull Request #20936 (efimpoberezkin)