[SPARK-3503] Disable thread local cache in PooledByteBufAllocator

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Shuffle, Spark Core
• Target Version/s: 1.2.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 12/Sep/14 05:21
• Updated: 17/May/20 18:30
• Resolved: 04/Nov/14 08:40

Description

Feedback from Norman.

Thread-local caches should be disabled because the ByteBufs are allocated by the event loop thread, but released by the executor thread rather than the event loop thread. Those thread-local caches actually delay the recycling of buffers, leading to larger memory usage.

Issue Links

• links to: Github pull request