[SPARK-38759] Add StreamingQueryListener support in PySpark

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: PySpark, Structured Streaming
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 01/Apr/22 10:07
• Updated: 12/Dec/22 18:10
• Resolved: 05/Apr/22 01:50

Description

PySpark currently does not have the support of StreamingQueryListener in PySpark whereas DStream has it. This feature is important especially with Dataset.observe so users can monitor what's going on in their queries.

Issue Links

• fixes: SPARK-30105 Add StreamingQueryListener support to PySpark
• links to: [Github] Pull Request #36038 (HyukjinKwon)