[SPARK-25756] pyspark pandas_udf does not respect append outputMode in structured streaming

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.2
• Component/s: PySpark, Structured Streaming
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Jan Bols
• Created: 17/Oct/18 11:11
• Updated: 12/Dec/22 18:11
• Resolved: 25/May/21 01:38

Description

When using the following setup:

• structured streaming
• a watermark and groupBy followed by an apply using a pandas grouped map udf
• a sink using an append outputMode

I would expect the following:

• udf to be called for each group --> OK
• when new data arrives, the udf will be called again –> OK
• when new data arrives for the same group, the udf will be called with the complete pandas dataframe of all received data for that group (up till the watermark) --> NOK: within the same group, the size of the pandas dataframe can decrease between invocations
• the results are only written to the sink once the processing time is passed the watermark --> NOK: every time the udf is called, new results are being sent to the output

It looks like pandas udf is unusable for structured streaming this way.