Details
-
Wish
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
2.2.0
-
None
Description
When you're using the watermark feature, you're forced to provide an absolute duration to identify late events. For the case we're using structured streaming for, this is not completely working. In our case, late events will be possible on the next business day. So I'd like to use 24 hours watermark for Sunday-Thursday, 72 hours for Friday, 48 for Saturday. What I would suggest is being able to use a function or expression to withWatermark so people can implement this or similar behaviors. If this sounds like a good idea, I can probably supply a pull request.