[SPARK-24036] Stateful operators in continuous processing

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Jose Torres
• Created: 20/Apr/18 18:10
• Updated: 16/Mar/20 22:53

Description

The first iteration of continuous processing in Spark 2.3 does not work with stateful operators.

Issue Links

• links to: [Github] Pull Request #21353 (xuanyuanking)

Sub-Tasks

• refactor continuous write exec to its own class: Resolved
• remove restarting iterators hack: Resolved
• support single partition aggregates: Resolved
• add flag to remove whitelist of continuous processing operators: Resolved
• create the bottom-of-task RDD with row buffer: Resolved
• continuous shuffle dependency: Resolved
• create the top-of-task RDD sending rows to the remote buffer: Resolved
• continuous replacement for ShuffleExchangeExec: Resolved
• Scheduler changes for continuous processing shuffle support: Resolved
• implement continuous processing coalesce(1): Resolved
• TCP based shuffle: Resolved