[SPARK-14367] spark.memory.useLegacyMode=true in 1.6 does not yield the same memory behavior as Spark 1.3

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Not A Bug
• Affects Version/s: 1.6.0
• Component/s: Spark Core
• Labels: backwards-compatibility

Environment

Ubuntu 15.10 with ibm-java-ppc64le-80.

• Assignee: Unassigned
• Reporter: Tom Hubregtsen
• Created: 04/Apr/16 13:57
• Updated: 11/Apr/16 18:07
• Resolved: 11/Apr/16 18:07

Description

Hi,

I am trying to get the same memory behavior in Spark 1.6 as I had in Spark 1.3 with default settings.

I set
-driver-java-options "-Dspark.memory.useLegacyMode=true -Dspark.shuffle.memoryFraction=0.2 -Dspark.storage.memoryFraction=0.6 -Dspark.storage.unrollFraction=0.2"
in Spark 1.6.

But the numbers don't add up. For instance:
--driver-java-options "-Dspark.shuffle.memoryFraction=0.1 -Dspark.storage.memoryFraction=0.1"
in Spark 1.3.1 leads to:
16/03/29 14:47:36 INFO MemoryStore: MemoryStore started with capacity 46.1 MB
The same in Spark 1.6.0 with -Dspark.memory.useLegacyMode=true -Dspark.shuffle.memoryFraction=0.1 -Dspark.storage.memoryFraction=0.1.
16/03/29 14:50:55 INFO MemoryStore: MemoryStore started with capacity 92.2 MB

If I then increase both fractions to 0.2, the numbers of the MemoryStore both double (as one would expect), but that means there is still a 2x difference in allocated memory between Spark 1.3 and Spark 1.6. So my question:

I believe a parameter that reads
spark.memory.useLegacyMode=true
should yield the exact memory behavior as in the Legacy version.