[SPARK-32293] Inconsistent default unit between Spark memory configs and JVM option

• Type: Bug
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.2.1, 2.2.2, 2.2.3, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.3.4, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 2.4.6, 3.0.0, 3.0.1, 3.1.0
• Component/s: Documentation, Spark Core
• Assignee: Unassigned
• Reporter: Attila Zsolt Piros
• Created: 13/Jul/20 18:11
• Updated: 03/Aug/20 11:38

Description

Spark's maximum memory can be configured in several ways:

• via Spark config
• command line argument
• environment variables

Both for executors and for the driver the memory can be configured separately. All of these are following the format of JVM memory configurations in a way they are using the very same size unit suffixes ("k", "m", "g" or "t") but there is an inconsistency regarding the default unit. When no suffix is given then the given amount is passed as it is to the JVM (to the -Xmx and -Xms options) where this memory options are using bytes as a default unit, for this please see the example here:

The following examples show how to set the maximum allowed size of allocated memory to 80 MB using various units:

-Xmx83886080 
-Xmx81920k 
-Xmx80m

Although the Spark memory config default suffix unit is "m".

Issue Links

• links to: [Github] Pull Request #29090 (attilapiros)