[SPARK-27833] java.lang.AssertionError: assertion failed: No plan for EventTimeWatermark

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: Structured Streaming
• Labels: bulk-closed spark-streaming-kafka

Environment

spark 2.3.0
java 1.8
kafka version 0.10.

• Assignee: Unassigned
• Reporter: Raviteja
• Created: 24/May/19 13:02
• Updated: 25/May/21 01:49
• Resolved: 25/May/21 01:41

Description

Hi ,
We have a requirement to read data from kafka, apply some transformation and store data to database .For this we are implementing watermarking feature along with aggregate function and for storing we are writing our own sink (Structured streaming) .we are using spark 2.3.0, java 1.8 and kafka version 0.10.
We are getting the below error.

"java.lang.AssertionError: assertion failed: No plan for EventTimeWatermark timestamp#39: timestamp, interval 2 minutes"
 
works perfectly fine when we use Console as sink instead custom sink. For Debugging the issue, we are performing "dataframe.show()" in our custom sink and nothing else.

Please find the attachment for the Error log and the code. Please look into this issue as this a blocker and we are not able to proceed further or find any alternatives as we need watermarking feature.