[SPARK-38965] Optimize RemoteBlockPushResolver with a memory pool

• Type: Bug
• Status: In Progress
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: Shuffle
• Assignee: Unassigned
• Reporter: Wan Kun
• Created: 20/Apr/22 08:24
• Updated: 25/Apr/22 10:22

Description

For push-based shuffle service, there are many BLOCK_APPEND_COLLISION_DETECTED when there are many small map tasks outputs. In RemoteBlockPushResolver, if one map task pushed blocks is writing, the others map tasks pushed blocks will failed in onComplete() method.
And RemoteBlockPushResolver has no memory limit , so many executors will OOM when there are many small pushed blocks waiting to be written to the final data file.

Issue Links

• links to: [Github] Pull Request #36279 (wankunde)