[SPARK-5841] Memory leak in DiskBlockManager

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.1
• Fix Version/s: 1.3.0
• Component/s: Block Manager, Spark Core
• Assignee: Matt Whelan
• Reporter: Matt Whelan
• Created: 16/Feb/15 19:29
• Updated: 17/May/20 18:20
• Resolved: 16/Feb/15 22:54

Description

DiskBlockManager registers a Runtime shutdown hook, which creates a hard reference to the entire Driver ActorSystem. If a long-running JVM repeatedly creates and destroys SparkContext instances, it leaks memory.

I suggest we deregister the shutdown hook if DiskBlockManager.stop is called. It's redundant at that point.

PR coming.

See also http://mail-archives.apache.org/mod_mbox/spark-user/201501.mbox/%3CCA+kjH+w_DDTEBE9XB6NrPxLTUXD=[e-mail]%3E

Issue Links

• is duplicated by: SPARK-5869 Exception when deleting Spark local dirs when shutting down DiskBlockManager
• is duplicated by: SPARK-8024 Luigi triggering resolved Blockmanager bug
• links to: [Github] Pull Request #4627 (MattWhelan)
• links to: [Github] Pull Request #4648 (srowen)
• links to: [Github] Pull Request #4690 (nishkamravi2)