[SPARK-38078] Aggregation with Watermark in AppendMode is holding data beyong water mark boundary.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 3.2.0
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: krishna
• Created: 31/Jan/22 16:11
• Updated: 08/Apr/23 22:01
• Resolved: 08/Apr/23 22:01

Description

 I am struggling with a unique issue. I am not sure if my understanding is wrong or this is a bug with spark.
 

1.  I am reading a stream from events hub/kafka ( Extract)
2.  Pivoting and Aggregating the above dataframe ( Transformation). This is a WATERMARKED aggregation.
3.  writing the aggregation to Console/Delta table in APPEND  mode with a Trigger . 

However, the most recently published message to event hub is not writing to console/delta even after falling out of the watermark time. 
 
 My understanding is the event should be inserted to  the Delta table after Eventtime+Watermark.
 

Moreover, all the events in the memory stored must be flushed out to the sink irrespective of the watermark before stopping to mark a graceful shutdown .

 

Please advise.