[SPARK-4808] Spark fails to spill with small number of large objects

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.2, 1.1.0, 1.2.0, 1.2.1
• Fix Version/s: 1.4.0
• Component/s: Spark Core
• Target Version/s: 1.4.0
• Assignee: Mingyu Kim
• Reporter: Dennis Lawler
• Created: 10/Dec/14 01:50
• Updated: 19/May/15 16:11
• Resolved: 19/May/15 16:11

Description

Spillable's maybeSpill does not allow spill to occur until at least 1000 elements have been spilled, and then will only evaluate spill every 32nd element thereafter. When there is a small number of very large items being tracked, out-of-memory conditions may occur.

I suspect that this and the every-32nd-element behavior was to reduce the impact of the estimateSize() call. This method was extracted into SizeTracker, which implements its own exponential backup for size estimation, so now we are only avoiding using the resulting estimated size.

Issue Links

• relates to: SPARK-5915 Spillable should check every N bytes rather than every 32 elements
• links to: [Github] Pull Request #3656 (lawlerd)
• links to: [Github] Pull Request #4420 (mingyukim)