[SPARK-6190] create LargeByteBuffer abstraction for eliminating 2GB limit on blocks

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: Spark Core
• Assignee: Josh Rosen Assign to me
• Reporter: Imran Rashid
• Created: 05/Mar/15 18:08
• Updated: 13/Feb/19 18:37
• Resolved: 16/May/18 18:07

Description

A key component in eliminating the 2GB limit on blocks is creating a proper abstraction for storing more than 2GB. Currently spark is limited by a reliance on nio ByteBuffer and netty ByteBuf, both of which are limited at 2GB. This task will introduce the new abstraction and the relevant implementation and utilities, without effecting the existing implementation at all.

Issue Links

• is related to: SPARK-5928 Remote Shuffle Blocks cannot be more than 2 GB
• is related to: SPARK-3151 DiskStore attempts to map any size BlockId without checking MappedByteBuffer limit
• is related to: SPARK-1476 2GB limit in spark for blocks
• is related to: SPARK-1391 BlockManager cannot transfer blocks larger than 2G in size
• links to: [Github] Pull Request #5400 (squito)
• links to: GitHub Pull Request #5400