[SPARK-18234] Update mode in structured streaming

• Type: New Feature
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.1.1, 2.2.0
• Component/s: Structured Streaming
• Target Version/s: 2.2.0
• Assignee: Tathagata Das
• Reporter: Michael Armbrust
• Created: 02/Nov/16 22:33
• Updated: 22/Dec/16 16:49
• Resolved: 22/Dec/16 00:43

Description

We have this internal, but we should nail down the semantics and expose it to users. The idea of update mode is that any tuple that changes will be emitted. Open questions:

• do we need to reason about the keys for a given stream? For things like the foreach sink its up to the user. However, for more end to end use cases such as a JDBC sink, we need to know which row downstream is being updated.
• okay to not support files?

Issue Links

• links to: [Github] Pull Request #16360 (tdas)