We use Spark exclusively with StorageLevel.DiskOnly as our workloads are very sensitive to memory usage. Recently, we've spotted that the jobs sometimes OOM leaving lots of byte arrays on the heap. Upon further investigation, we've found that the arrays come from BlockManager.getRemoteBytes, which calls BlockTransferService.fetchBlockSync, which in its turn would allocate an on-heap ByteBuffer of the same size as the block (e.g. full partition), if the block was successfully retrieved over the network.
This memory is not accounted towards Spark storage/execution memory and could potentially lead to OOM if BlockManager fetches too many partitions in parallel. I wonder if this is intentional behaviour, or in fact a bug?