[SPARK-18908] It's hard for the user to see the failure if StreamExecution fails to create the logical plan

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.1.1, 2.2.0
• Component/s: Structured Streaming
• Target Version/s: 2.1.1
• Assignee: Shixiong Zhu Assign to me
• Reporter: Shixiong Zhu
• Created: 17/Dec/16 00:45
• Updated: 17/Jan/17 22:09
• Resolved: 22/Dec/16 06:03

Description

If the logical plan fails to create, e.g., some Source options are invalid, the user cannot use the code to detect the failure. The only place receiving this error is Thread's UncaughtExceptionHandler.

This bug is because logicalPlan is lazy, and when we try to create StreamingQueryException to wrap the exception thrown by creating logicalPlan, it calls logicalPlan agains.

Issue Links

• contains: SPARK-18907 Fix flaky test: o.a.s.sql.streaming.FileStreamSourceSuite max files per trigger - incorrect values
• links to: [Github] Pull Request #16322 (zsxwing)