[SPARK-24234] create the bottom-of-task RDD with row buffer

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: Structured Streaming
• Assignee: Jose Torres
• Reporter: Jose Torres
• Created: 10/May/18 01:21
• Updated: 13/Jul/18 05:26
• Resolved: 21/May/18 19:58

Description

https://docs.google.com/document/d/1IL4kJoKrZWeyIhklKUJqsW-yEN7V7aL05MmM65AYOfE/edit#heading=h.8t3ci57f7uii

 

This probably ought to be an abstraction of ContinuousDataSourceRDD and ContinuousQueuedDataReader. These classes do pretty much exactly what's needed, except the buffer is filled from the remote data source instead of a remote Spark task.

Issue Links

• links to: [Github] Pull Request #21337 (jose-torres)
• links to: [Github] Pull Request #21385 (jose-torres)