[SPARK-31265] Add -XX:MaxDirectMemorySize jvm options in yarn mode

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Invalid
• Affects Version/s: 3.0.0
• Component/s: Spark Core, YARN
• Assignee: Unassigned
• Reporter: wang-zhun
• Created: 26/Mar/20 06:22
• Updated: 17/May/20 18:14
• Resolved: 31/Mar/20 18:38

Description

Current memory composition `amMemory` + `amMemoryOverhead`

val capability = Records.newRecord(classOf[Resource])
capability.setMemory(amMemory + amMemoryOverhead)
capability.setVirtualCores(amCores)
if (amResources.nonEmpty) {
 ResourceRequestHelper.setResourceRequests(amResources, capability)
}
logDebug(s"Created resource capability for AM request: $capability")

 // Add Xmx for AM memory 
javaOpts += "-Xmx" + amMemory + "m"

It is possible that the physical memory of the container exceeds the limit and is killed by yarn.
I suggest setting `-XX:MaxDirectMemorySize` here

// Add Xmx for AM memory
javaOpts += "-Xmx" + amMemory + "m"
javaOpts += s"-XX:MaxDirectMemorySize=${amMemoryOverhead}m"

Issue Links

• links to: GitHub Pull Request #28069