[SPARK-24525] Provide an option to limit MemorySink memory usage

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.1
• Fix Version/s: 2.4.0
• Component/s: Structured Streaming
• Assignee: Mukul Murthy
• Reporter: Mukul Murthy
• Created: 11/Jun/18 22:41
• Updated: 26/Jun/18 22:53
• Resolved: 15/Jun/18 20:59

Description

MemorySink stores stream results in memory and is mostly used for testing and displaying streams, but for large streams, this can OOM the driver. We should add an option to limit the number of rows and the total size of a memory sink and not add any new data once either limit is hit. 

Issue Links

• is superceded by: SPARK-24662 Structured Streaming should support LIMIT
• links to: [Github] Pull Request #21559 (mukulmurthy)