[SPARK-16454] Consider adding a per-batch transform for structured streaming

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: Structured Streaming
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Holden Karau
• Created: 08/Jul/16 23:24
• Updated: 21/May/19 04:14
• Resolved: 21/May/19 04:14

Description

The new structured streaming API lacks the DStream functionality of transform (which allowed one to mix in existing RDD transformation logic). It would be useful to be able to do per-batch (even without any specific gaurantees about the batch being complete provided you eventually get called with the "catch up" records) processing as was done in the DStream API.

This might be useful for implementing Streaming Machine Learning on Structured Streaming.

Issue Links

• relates to: SPARK-16424 Add support for Structured Streaming to the ML Pipeline API