[SPARK-1476] 2GB limit in spark for blocks

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Duplicate
• Component/s: Spark Core

Environment

all

• Assignee: Unassigned
• Reporter: Mridul Muralidharan
• Created: 12/Apr/14 06:29
• Updated: 18/Dec/19 02:38
• Resolved: 26/Jun/15 17:15

Description

The underlying abstraction for blocks in spark is a ByteBuffer : which limits the size of the block to 2GB.
This has implication not just for managed blocks in use, but also for shuffle blocks (memory mapped blocks are limited to 2gig, even though the api allows for long), ser-deser via byte array backed outstreams (SPARK-1391), etc.

This is a severe limitation for use of spark when used on non trivial datasets.

Issue Links

• breaks: SPARK-1353 IllegalArgumentException when writing to disk
• depends upon: SPARK-1391 BlockManager cannot transfer blocks larger than 2G in size
• is duplicated by: SPARK-1353 IllegalArgumentException when writing to disk
• relates to: SPARK-6190 create LargeByteBuffer abstraction for eliminating 2GB limit on blocks
• relates to: SPARK-3151 DiskStore attempts to map any size BlockId without checking MappedByteBuffer limit
• requires: SPARK-5928 Remote Shuffle Blocks cannot be more than 2 GB
• requires: SPARK-1391 BlockManager cannot transfer blocks larger than 2G in size