[SPARK-45957] SQL on streaming Temp view fails

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Connect, Structured Streaming
• Labels: pull-request-available
• Epic Link: Streaming Spark Connect
• Assignee: Jean-Francois Desjeans Gauthier
• Reporter: Raghu Angadi
• Created: 16/Nov/23 18:59
• Updated: 27/Nov/23 23:32
• Resolved: 27/Nov/23 23:32

Description

The following code fails in the last step with Spark Connect.

The root cause is that Connect server triggers physical plan on a streaming Dataframe in SparkConnectPlanner.scala. Better to avoid that entirely, but at least for streaming it should be avoided since it cannot be done with a batch execution engine.

df = spark.readStream.format("rate").option("numPartitions", "1").load()
df.createOrReplaceTempView("temp_view")

view_df = spark.sql("SELECT * FROM temp_view") // FAILS

 

Issue Links

• links to: GitHub Pull Request #43845
• links to: GitHub Pull Request #43851