[SPARK-1042] spark cleans all java broadcast variables when it hits the spark.cleaner.ttl

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.8.0, 0.8.1, 0.9.0
• Fix Version/s: 0.9.2
• Component/s: Java API, Spark Core
• Labels: memory_leak
• Assignee: OuyangJin
• Reporter: Tal Sliwowicz
• Created: 23/Jan/14 09:29
• Updated: 21/Oct/14 06:53
• Resolved: 21/Oct/14 06:53

Description

When setting spark.cleaner.ttl, spark performs the cleanup on time - but it cleans all broadcast variables, not just the ones that are older than the ttl. This creates an exception when the next mapPartitions runs because it cannot find the broadcast variable, even when it was created immediately before running the task.

Our temp workaround - not set the ttl and suffer from an ongoing memory leak (forces a restart).

We are using JavaSparkContext and our broadcast variables are Java HashMaps.