[SPARK-41040] Self-union streaming query may fail when using readStream.table

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Structured Streaming
• Assignee: Shixiong Zhu
• Reporter: Shixiong Zhu
• Created: 08/Nov/22 05:40
• Updated: 08/Nov/22 16:33
• Resolved: 08/Nov/22 16:32

Description

In a self-union query, the batch plan created by the source will be shared by multiple nodes in the plan. When we transform the plan, the batch plan will be visited multiple times. Hence, the first visit will set the CatalogTable and the second visit will try to set it again and fail the query at this line.

Issue Links

• is broken by: SPARK-39564 Expose the information of catalog table to the logical plan in streaming query
• links to: [Github] Pull Request #38553 (zsxwing)