Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.4.1, 3.5.0
Description
Streaming queries do not support current_date() inside CollectMetrics. The primary reason is that current_date() (resolves to CurrentBatchTimestamp) is marked as non-deterministic. However, current_date and current_timestamp are both deterministic today, and current_batch_timestamp should be the same.
As an example, the query below fails due to observe call on the DataFrame.
val inputData = MemoryStream[Timestamp]
inputData.toDF()
.filter("value < current_date()")
.observe("metrics", count(expr("value >= current_date()")).alias("dropped"))
.writeStream
.queryName("ts_metrics_test")
.format("memory")
.outputMode("append")
.start()
Attachments
Issue Links
- links to