[SPARK-17286] The fetched data of Shuffle read stored in the ManagedBuffer, so its underlying data stored in off-memory or file?

• Type: Question
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Component/s: Block Manager, Input/Output, Shuffle, Spark Core
• Assignee: Unassigned
• Reporter: song fengfei
• Created: 29/Aug/16 05:48
• Updated: 17/May/20 18:21
• Resolved: 29/Aug/16 09:17

Description

The fetched data of Shuffle read stored in the ManagedBuffer, so its underlying data stored in off-memory or file? if stored in off-memory, Whether it is also managed by spark memorymanager ? and if a map output is too big, placed it directly in the memory isn't very easily lead to oom? I did not understand this piece, hoping to get the answer here,thanks!