[SPARK-30246] Spark on Yarn External Shuffle Service Memory Leak

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.3
• Fix Version/s: 2.4.5, 3.0.0
• Component/s: Shuffle, Spark Core

Environment

hadoop 2.7.3
spark 2.4.3
jdk 1.8.0_60

• Assignee: Henrique dos Santos Goulart
• Reporter: uncle-huang
• Created: 13/Dec/19 04:47
• Updated: 15/Feb/20 08:29
• Resolved: 15/Jan/20 21:36

Description

In our large busy yarn cluster which deploy Spark external shuffle service as part of YARN NM aux service, we encountered OOM in some NMs.
after i dump the heap memory and found there are some StremState objects still in heap, but the app which the StreamState belongs to is already finished.

Here is some relate Figures:

The heap dump below shows that the memory consumption mainly consists of two parts:
(1) OneForOneStreamManager (4,429,796,424 (77.11%) bytes)
(2) PoolChunk(occupy 1,059,201,712 (18.44%) bytes. )

dig into the OneForOneStreamManager, there are some StreaStates still remained :

incomming references to StreamState::associatedChannel:

Issue Links

• links to: GitHub Pull Request #27060
• links to: GitHub Pull Request #27064
• links to: GitHub Pull Request #27201