[SPARK-39564] Expose the information of catalog table to the logical plan in streaming query

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 23/Jun/22 02:28
• Updated: 08/Nov/22 16:32
• Resolved: 27/Jun/22 20:48

Description

In Spark 3.1.0, we added ability to read and write from/to catalog table for streaming query. (Read: SPARK-32885 / Write: SPARK-32896)

When the streaming query is being executed with such API, the table identifier for source and sink is "conditionally" exposed to the logical plan, depending on the DSv1 vs DSv2 and source vs sink.

Since this is the only way to get the information of table, we would like to make some changes on relevant logical nodes to carry out and expose the information of table, specifically, table identifier.

 

 

Issue Links

• breaks: SPARK-41040 Self-union streaming query may fail when using readStream.table
• links to: [Github] Pull Request #36963 (HeartSaVioR)
• links to: [Github] Pull Request #36963 (HeartSaVioR)
• links to: [Github] Pull Request #37042 (HeartSaVioR)