[SPARK-29426] Watermark does not take effect

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.4.3
• Component/s: Structured Streaming
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: jingshanglu
• Created: 10/Oct/19 10:11
• Updated: 25/May/21 01:54
• Resolved: 25/May/21 01:41

Description

I use withWatermark and window to express windowed aggregations, but the Watermark does not take effect.

my code:

// code placeholder
Dataset<Row> clientSqlIpCount = mes.withWatermark("timestamp","1 minute")
        .groupBy(
                functions.window(mes.col("timestamp"),"10 minutes","5 minutes"),
                mes.col("sql"),mes.col("client"),mes.col("ip"))
        .count();
StreamingQuery query = clientSqlIpCount
                .writeStream()
                .outputMode("Update")
                .format("console")
                .start();
spark.streams().awaitAnyTermination();