[SPARK-13566] Deadlock between MemoryStore and BlockManager

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 1.6.2
• Component/s: Block Manager, Spark Core

Environment

Spark 1.6.0 hadoop2.2.0 jdk1.8.0_65 centOs 6.2

• Target Version/s: 1.6.2
• Assignee: cen yuhai
• Reporter: cen yuhai
• Created: 29/Feb/16 12:46
• Updated: 17/Oct/16 11:13
• Resolved: 06/May/16 20:51

Description

===================================================
"block-manager-slave-async-thread-pool-1":
at org.apache.spark.storage.MemoryStore.remove(MemoryStore.scala:216)

• waiting to lock <0x00000005895b09b0> (a org.apache.spark.memory.UnifiedMemoryManager)
  at org.apache.spark.storage.BlockManager.removeBlock(BlockManager.scala:1114)
• locked <0x000000058ed6aae0> (a org.apache.spark.storage.BlockInfo)
  at org.apache.spark.storage.BlockManager$$anonfun$removeBroadcast$2.apply(BlockManager.scala:1101)
  at org.apache.spark.storage.BlockManager$$anonfun$removeBroadcast$2.apply(BlockManager.scala:1101)
  at scala.collection.immutable.Set$Set2.foreach(Set.scala:94)
  at org.apache.spark.storage.BlockManager.removeBroadcast(BlockManager.scala:1101)
  at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$4.apply$mcI$sp(BlockManagerSlaveEndpoint.scala:65)
  at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$4.apply(BlockManagerSlaveEndpoint.scala:65)
  at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$4.apply(BlockManagerSlaveEndpoint.scala:65)
  at org.apache.spark.storage.BlockManagerSlaveEndpoint$$anonfun$1.apply(BlockManagerSlaveEndpoint.scala:84)
  at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
  at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)
  "Executor task launch worker-10":
  at org.apache.spark.storage.BlockManager.dropFromMemory(BlockManager.scala:1032)
• waiting to lock <0x000000059a0988b8> (a org.apache.spark.storage.BlockInfo)
  at org.apache.spark.storage.BlockManager.dropFromMemory(BlockManager.scala:1009)
  at org.apache.spark.storage.MemoryStore$$anonfun$evictBlocksToFreeSpace$2.apply(MemoryStore.scala:460)
  at org.apache.spark.storage.MemoryStore$$anonfun$evictBlocksToFreeSpace$2.apply(MemoryStore.scala:449)
  at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

Issue Links

• is duplicated by: SPARK-15558 Deadlock when retreiving shuffled cached data
• is duplicated by: SPARK-15943 Spark driver hangs up periodically (cannot receive any reply in 120 seconds)
• is duplicated by: SPARK-16513 Spark executor deadlocks itself in memory management
• is duplicated by: SPARK-16564 DeadLock happens when ‘StaticMemoryManager‘ release in-memory block
• links to: [Github] Pull Request #11546 (cenyuhai)