[SYSTEMDS-1922] Memory management spark broadcasts

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: SystemML 1.0.0
• Assignee: Matthias Boehm
• Reporter: Matthias Boehm
• Created: 20/Sep/17 21:50
• Updated: 30/Nov/17 22:25
• Resolved: 30/Nov/17 22:25

Description

So far, we assumed that spark's broadcast simply blockifies the given broadcast object (which creates a deep copy). However, additionally also holds a strong reference to the passed object which causes unnecessary memory overhead (and thus OOMs) when the passed object is later evicted.

This task aims to remove this unnecessary memory overhead and fix the used memory estimates for broadcast creation.