[SPARK-18811] Stream Source resolution should happen in StreamExecution thread, not main thread

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.2, 2.1.0
• Fix Version/s: 2.1.0
• Component/s: Structured Streaming
• Target Version/s: 2.1.0
• Assignee: Burak Yavuz
• Reporter: Burak Yavuz
• Created: 09/Dec/16 20:38
• Updated: 15/Dec/16 05:05
• Resolved: 10/Dec/16 06:50

Description

When you start a stream, if we are trying to resolve the source of the stream, for example if we need to resolve partition columns, this could take a long time. This long execution time should not block the main thread where `query.start()` was called on. It should happen in the stream execution thread possibly before starting any triggers.

Issue Links

• links to: [Github] Pull Request #16238 (brkyvz)