[SPARK-24039] remove restarting iterators hack

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: Structured Streaming
• Assignee: Jose Torres
• Reporter: Jose Torres
• Created: 20/Apr/18 18:15
• Updated: 13/Jul/18 05:27
• Resolved: 04/May/18 21:15

Description

Currently, continuous processing execution calls next() to restart the query iterator after it returns false. This doesn't work for complex RDDs - we need to call compute() instead.

This isn't refactoring-only; changes will be required to keep the reader from starting over in each compute() call.

Issue Links

• links to: [Github] Pull Request #21200 (jose-torres)