[SPARK-34321] Fix the guarantee of foreachBatch

• Type: Documentation
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 3.2.0
• Component/s: Structured Streaming
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 02/Feb/21 05:43
• Updated: 02/Feb/21 05:48
• Resolved: 02/Feb/21 05:44

Description

Similar to SPARK-28650, foreachBatch API document also documents the guarantee:

The batchId can be used to deduplicate and transactionally write the output (that is, the provided Dataset) to external systems. The output Dataset is guaranteed to be exactly the same for the same batchId

But like the reason of fixing the document of ForeachWriter in SPARK-28650, it is not hard to break the guarantee by changing the partition number.