[SPARK-22278] Expose current event time watermark and current processing time in GroupState

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: Structured Streaming
• Target Version/s: 2.3.0
• Assignee: Tathagata Das
• Reporter: Tathagata Das
• Created: 14/Oct/17 00:15
• Updated: 14/Jan/18 07:56
• Resolved: 18/Oct/17 03:09

Description

Complex state-updating and/or timeout-handling logic in mapGroupsWithState functions may require taking decisions based on the current event-time watermark and/or processing-time. Currently, you can use the sql function `current_timestamp` to get the current processing time, but it needs to passed inserted in every row with a select, and then passed through the encoder, which isnt efficient. Furthermore, there is no way to get the current watermark.

This JIRA is to expose them through the GroupState API.

Issue Links

• links to: [Github] Pull Request #19495 (tdas)