[SPARK-46873] PySpark spark.streams should not recreate new StreamingQueryManager

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Connect, PySpark, SS
• Labels: pull-request-available
• Assignee: Wei Liu
• Reporter: Wei Liu
• Created: 26/Jan/24 06:35
• Updated: 28/Jan/24 23:53
• Resolved: 28/Jan/24 23:53

Description

In Scala, there is only one streaming query manager for one spark session:

```

scala> spark.streams

val res0: org.apache.spark.sql.streaming.StreamingQueryManager = org.apache.spark.sql.streaming.StreamingQueryManager@46bb8cba

 

scala> spark.streams

val res1: org.apache.spark.sql.streaming.StreamingQueryManager = org.apache.spark.sql.streaming.StreamingQueryManager@46bb8cba

 

scala> spark.streams

val res2: org.apache.spark.sql.streaming.StreamingQueryManager = org.apache.spark.sql.streaming.StreamingQueryManager@46bb8cba

 

scala> spark.streams

val res3: org.apache.spark.sql.streaming.StreamingQueryManager = org.apache.spark.sql.streaming.StreamingQueryManager@46bb8cba

```

 

In Python, this is currently false:

```

>>> spark.streams

<pyspark.sql.connect.streaming.query.StreamingQueryManager object at 0x1011f7c10>

>>> spark.streams

<pyspark.sql.connect.streaming.query.StreamingQueryManager object at 0x1011f71f0>

>>> spark.streams

<pyspark.sql.connect.streaming.query.StreamingQueryManager object at 0x1011f7be0>

>>> spark.streams

<pyspark.sql.connect.streaming.query.StreamingQueryManager object at 0x1011f7c40>

```

 

Python should align scala behavior. 

Issue Links

• links to: GitHub Pull Request #44898