[SPARK-47698] Current doc section handling-late-data-and-watermarking is misleading after the support for multiple stateful operators

• Type: Documentation
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: Documentation
• Assignee: Unassigned
• Reporter: Lingeshwaran Radhakrishnan
• Created: 02/Apr/24 19:04
• Updated: 02/Apr/24 19:04

Description

This section of the doc which explains the nuances of handling late records using beautiful diagrams has gone out of context and a bit misleading after multiple stateful operators' support was introduced with https://issues.apache.org/jira/browse/SPARK-40925 

 

Previously watermark is applied for batch N to filter out inputs in batch N. With support for multiple stateful operators, the watermark is applied for batch N-1 instead. The doc section above should reflect this new behavior to avoid confusion.