[SPARK-19774] StreamExecution should call stop() on sources when a stream fails

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.2, 2.1.0
• Fix Version/s: 2.1.1, 2.2.0
• Component/s: Structured Streaming
• Target Version/s: 2.2.0
• Assignee: Burak Yavuz
• Reporter: Burak Yavuz
• Created: 28/Feb/17 22:02
• Updated: 03/Mar/17 18:35
• Resolved: 03/Mar/17 18:35

Description

We call stop() on a Structured Streaming Source only when the stream is shutdown when a user calls streamingQuery.stop(). We should actually stop all sources when the stream fails as well, otherwise we may leak resources, e.g. connections to Kafka.

Issue Links

• links to: [Github] Pull Request #17107 (brkyvz)