[SPARK-32500] Query and Batch Id not set for Structured Streaming Jobs in case of ForeachBatch in PySpark

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.6
• Component/s: PySpark, Structured Streaming
• Assignee: Unassigned
• Reporter: Abhishek Dixit
• Created: 30/Jul/20 15:33
• Updated: 12/Dec/22 18:10

Description

Query Id and Batch Id information is not available for jobs started by structured streaming query when foreachBatch API is used in PySpark.

This happens only with foreachBatch in pyspark. ForeachBatch in scala works fine, and also other structured streaming sinks in pyspark work fine. I am attaching a screenshot of jobs pages.

I think job group is not set properly when foreachBatch is used via pyspark. I have a framework that depends on the queryId and batchId information available in the job properties and so my framework doesn't work for pyspark-foreachBatch use case.