[SPARK-10815] Public API: Streaming Sources and Sinks

• Type: New Feature
• Status: Resolved
• Priority: Critical
• Resolution: Incomplete
• Component/s: Structured Streaming
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Reynold Xin
• Created: 24/Sep/15 22:07
• Updated: 21/May/19 04:13
• Resolved: 21/May/19 04:13

Description

The existing (in 2.0) source/sink interface for structured streaming depends on RDDs. This dependency has two issues:

1. The RDD interface is wide and difficult to stabilize across versions. This is similar to point 1 in https://issues.apache.org/jira/browse/SPARK-15689. Ideally, a source/sink implementation created for Spark 2.x should work in Spark 10.x, assuming the JVM is still around.

2. It is difficult to swap in/out a different execution engine.

The purpose of this ticket is to create a stable interface that addresses the above two.