[SPARK-26884] Let task acquire memory accurately when using spilled memory

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: SongYadong
• Created: 15/Feb/19 06:33
• Updated: 20/Feb/19 19:01
• Resolved: 20/Feb/19 07:20

Description

When task can't get required execution memory, it will call spill() of consumers to release more memory. After spilling, it tries again to acquire memory needed which is (required - got). That's not accurate as actually released memory by spilling may not equal to size it try to spill.
So it may be better to acquire memory in more accurate size :
1. when released >= needed, acquire needed size. (required - got)
2. when released < needed, acquire released size.

 

Issue Links

• links to: GitHub Pull Request #23794