[SPARK-20691] Difference between Storage Memory as seen internally and in web UI

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: Web UI
• Labels: bulk-closed starter
• Assignee: Unassigned
• Reporter: Jacek Laskowski
• Created: 10/May/17 08:05
• Updated: 08/Oct/19 05:42
• Resolved: 08/Oct/19 05:42

Description

I set Major priority as it's visible to a user.

There's a difference in what the size of Storage Memory is managed internally and displayed to a user in web UI.

I found it while answering How does web UI calculate Storage Memory (in Executors tab)? on StackOverflow.

In short (quoting the main parts), when you start a Spark app (say spark-shell) you see 912.3 MB RAM for Storage Memory:

$ ./bin/spark-shell --conf spark.driver.memory=2g
...
17/05/07 15:20:50 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.1.8:57177 with 912.3 MB RAM, BlockManagerId(driver, 192.168.1.8, 57177, None)

but in the web UI you'll see 956.6 MB due to the way the custom JavaScript function formatBytes in utils.js calculates the value. That translates to the following Scala code:

def formatBytes(bytes: Double) = {
  val k = 1000
  val i = math.floor(math.log(bytes) / math.log(k))
  val maxMemoryWebUI = bytes / math.pow(k, i)
  f"$maxMemoryWebUI%1.1f"
}
scala> println(formatBytes(maxMemory))
956.6

Issue Links

• links to: [Github] Pull Request #18229 (mkesselaers)