[SPARK-22062] BlockManager does not account for memory consumed by remote fetches

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: Block Manager, Spark Core
• Assignee: Saisai Shao
• Reporter: Sergei Lebedev
• Created: 19/Sep/17 15:21
• Updated: 17/May/20 18:21
• Resolved: 17/Oct/17 14:55

Description

We use Spark exclusively with StorageLevel.DiskOnly as our workloads are very sensitive to memory usage. Recently, we've spotted that the jobs sometimes OOM leaving lots of byte[] arrays on the heap. Upon further investigation, we've found that the arrays come from BlockManager.getRemoteBytes, which calls BlockTransferService.fetchBlockSync, which in its turn would allocate an on-heap ByteBuffer of the same size as the block (e.g. full partition), if the block was successfully retrieved over the network.

This memory is not accounted towards Spark storage/execution memory and could potentially lead to OOM if BlockManager fetches too many partitions in parallel. I wonder if this is intentional behaviour, or in fact a bug?

Issue Links

• relates to: SPARK-6235 Address various 2G limits
• links to: [Github] Pull Request #19476 (jerryshao)