[SPARK-24063] Control maximum epoch backlog

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: Structured Streaming
• Assignee: Gabor Somogyi
• Reporter: Efim Poberezkin
• Created: 24/Apr/18 08:14
• Updated: 18/Mar/19 16:57
• Resolved: 27/Feb/19 17:53

Description

As pointed out by joseph.torres in https://github.com/apache/spark/pull/20936, both epoch queue and commits/offsets maps are unbounded by the number of waiting epochs. According to his proposal, we should introduce some configuration option for maximum epoch backlog and report an error if the number of waiting epochs exceeds it.

Issue Links

• links to: [Github] Pull Request #21392 (efimpoberezkin)
• links to: [Github] Pull Request #23156 (gaborgsomogyi)
• links to: GitHub Pull Request #23156