[SPARK-4772] Accumulators leak memory, both temporarily and permanently

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.0.3, 1.1.2, 1.2.1, 1.3.0
• Component/s: Spark Core
• Labels: accumulators
• Target Version/s: 1.0.3, 1.1.2, 1.2.1, 1.3.0
• Assignee: Nathan Kronenfeld
• Reporter: Nathan Kronenfeld
• Created: 05/Dec/14 21:49
• Updated: 17/Dec/14 20:19
• Resolved: 17/Dec/14 20:19

Description

Accumulators.localAccums is cleared at the beginning of a task, and not at the end.

This means that any locally accumulated values hang around until another task is run on that thread.

If for some reason, the thread dies, said values hang around indefinitely.

This is really only a big issue with very large accumulators.

Issue Links

• links to: [Github] Pull Request #3570 (nkronenfeld)