[SPARK-4030] `destroy` method in Broadcast should be public

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0, 1.2.0
• Fix Version/s: 1.2.0
• Component/s: Block Manager, Spark Core
• Assignee: Shivaram Venkataraman
• Reporter: Shivaram Venkataraman
• Created: 21/Oct/14 05:12
• Updated: 27/Oct/14 15:51
• Resolved: 27/Oct/14 15:51

Description

The destroy method in Broadcast.scala https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/broadcast/Broadcast.scala#L91 is right now marked as private[spark]

This prevents long-running applications from cleaning up memory used by broadcast variables on the driver. Also as broadcast variables are always created with persistence MEMORY_DISK, this slows down jobs when old broadcast variables are flushed to disk.

Making `destroy` public can help applications control the lifetime.

Issue Links

• relates to: SPARK-3885 Provide mechanism to remove accumulators once they are no longer used
• links to: [Github] Pull Request #2922 (shivaram)