Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25008

Add memory mode info to showMemoryUsage in TaskMemoryManager

    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 2.3.0
    • Fix Version/s: None
    • Component/s: Spark Core
    • Labels:

      Description

      TaskMemoryManager prints the current memory usage information before throwing an OOM exception which is helpful in debugging issues. This log does not have the memory mode information which can be also useful to quickly determine which memory users need to increase.

      This JIRA is to add that information to showMemoryUsage method of TaskMemoryManager.

      Current logs:

      18/07/03 17:57:16 INFO memory.TaskMemoryManager: Memory used in task 318
      18/07/03 17:57:16 INFO memory.TaskMemoryManager: Acquired by org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@7f084d1b: 1024.0 KB
      18/07/03 17:57:16 INFO memory.TaskMemoryManager: Acquired by org.apache.spark.shuffle.sort.ShuffleExternalSorter@713d50f2: 32.0 KB
      18/07/03 17:57:16 INFO memory.TaskMemoryManager: 0 bytes of memory were used by task 318 but are not associated with specific consumers
      18/07/03 17:57:16 INFO memory.TaskMemoryManager: 1081344 bytes of memory are used for execution and 306201016 bytes of memory are used for storage
      18/07/03 17:57:16 ERROR executor.Executor: Exception in task 86.0 in stage 49.0 (TID 318)
      java.lang.OutOfMemoryError: Unable to acquire 326284160 bytes of memory, got 3112960
       at org.apache.spark.memory.MemoryConsumer.allocatePage(MemoryConsumer.java:127)
       at org.apache.spark.shuffle.sort.ShuffleExternalSorter.acquireNewPageIfNecessary(ShuffleExternalSorter.java:359)
       at org.apache.spark.shuffle.sort.ShuffleExternalSorter.insertRecord(ShuffleExternalSorter.java:382)
       at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.insertRecordIntoSorter(UnsafeShuffleWriter.java:246)
       at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:167)
       at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
       at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
       at org.apache.spark.scheduler.Task.run(Task.scala:108)
       at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       at java.lang.Thread.run(Thread.java:745)
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ankur.gupta Ankur Gupta
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: