[SPARK-42626] Add Destructive Iterator for SparkResult

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.5.0
• Component/s: Connect
• Epic Link: Spark Connect Scala Client - Basic
• Assignee: Tengfei Huang
• Reporter: Herman van Hövell
• Created: 28/Feb/23 23:58
• Updated: 06/Jun/23 01:31
• Resolved: 06/Jun/23 01:31

Description

Add a destructive iterator to SparkResult. Instead of keeping everything in memory for the life time of SparkResult object, clean it up as soon as we know we are done with it. We can use this for Dataset.toLocalIterator.

Issue Links

• links to: https://github.com/apache/spark/pull/40610