[SPARK-22911] Migrate structured streaming sources to new DataSourceV2 APIs

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: Structured Streaming
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Jose Torres
• Created: 27/Dec/17 21:30
• Updated: 08/Oct/19 05:44
• Resolved: 08/Oct/19 05:44

Sub-Tasks

• Support v2 streaming sources and sinks in MicroBatchExecution: Resolved
• Migrate MemoryStream to V2: Resolved
• Migrate MemorySink fully to v2: Resolved
• Migrate Microbatch ConsoleSink to v2: Resolved
• Migrate rate source to v2: Resolved
• Migrate text socket source to v2: Resolved
• Migrate Kafka batch source to v2: Resolved
• Migrate Kafka microbatch source to v2: Resolved
• Migrate foreach sink: Resolved
• Migrate unit test sources: Resolved
• Migrate unit test sinks: Resolved
• Migrate kafka sink: Resolved
• Unify continuous and microbatch V2 sinks: Resolved
• design doc for write side: Resolved
• design doc for read side: Resolved
• clean up StreamWriter factory lifecycle: Resolved
• add epoch ID to data writer factory: Resolved
• make StreamWriter not a DataSourceWriter subclass: Resolved
• Continuous rate source uses different offset format: Resolved