[SPARK-38861] GroupState.getCurrentWatermarkMs() on static Dataframe throws exception but should return -1

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.1.2
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Trevor Christman
• Created: 11/Apr/22 14:17
• Updated: 16/Apr/22 19:12

Description

As per documentation GroupState.getCurrentWatermarkMs() returns -1 when operating on a static Dataframe, as found here: https://spark.apache.org/docs/3.1.3/api/scala/org/apache/spark/sql/streaming/GroupState.html#getCurrentWatermarkMs():Long

 

In reality this function throws UnsupportedOperationException as documented when operating on a DataFrame without a watermark set.  This happens even if the static DataFrame had a (useless) watermark configured.