[SPARK-29301] Removing block is not reflected to the driver/executor's storage memory

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.4.4, 3.0.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Jungtaek Lim
• Created: 30/Sep/19 06:29
• Updated: 16/Mar/20 07:25
• Resolved: 16/Mar/20 07:24

Description

While investigating SPARK-29055 I've found that heap memory in driver doesn't increase which would denote memory leak happen, but storage memory doesn't decrease even there're some broadcast blocks getting removed - these values keep increasing.

Please refer SPARK-29055 to see the step to reproduce. Very easy to see it, as running simple query repeatedly would reproduce the issue.

Issue Links

• is superceded by: SPARK-29055 Spark UI storage memory increasing overtime