[SPARK-25008] Add memory mode info to showMemoryUsage in TaskMemoryManager

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Ankur Gupta
• Created: 02/Aug/18 21:52
• Updated: 08/Oct/19 05:43
• Resolved: 08/Oct/19 05:43

Description

TaskMemoryManager prints the current memory usage information before throwing an OOM exception which is helpful in debugging issues. This log does not have the memory mode information which can be also useful to quickly determine which memory users need to increase.

This JIRA is to add that information to showMemoryUsage method of TaskMemoryManager.

Current logs:

18/07/03 17:57:16 INFO memory.TaskMemoryManager: Memory used in task 318
18/07/03 17:57:16 INFO memory.TaskMemoryManager: Acquired by org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@7f084d1b: 1024.0 KB
18/07/03 17:57:16 INFO memory.TaskMemoryManager: Acquired by org.apache.spark.shuffle.sort.ShuffleExternalSorter@713d50f2: 32.0 KB
18/07/03 17:57:16 INFO memory.TaskMemoryManager: 0 bytes of memory were used by task 318 but are not associated with specific consumers
18/07/03 17:57:16 INFO memory.TaskMemoryManager: 1081344 bytes of memory are used for execution and 306201016 bytes of memory are used for storage
18/07/03 17:57:16 ERROR executor.Executor: Exception in task 86.0 in stage 49.0 (TID 318)
java.lang.OutOfMemoryError: Unable to acquire 326284160 bytes of memory, got 3112960
 at org.apache.spark.memory.MemoryConsumer.allocatePage(MemoryConsumer.java:127)
 at org.apache.spark.shuffle.sort.ShuffleExternalSorter.acquireNewPageIfNecessary(ShuffleExternalSorter.java:359)
 at org.apache.spark.shuffle.sort.ShuffleExternalSorter.insertRecord(ShuffleExternalSorter.java:382)
 at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.insertRecordIntoSorter(UnsafeShuffleWriter.java:246)
 at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:167)
 at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
 at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
 at org.apache.spark.scheduler.Task.run(Task.scala:108)
 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)