[SPARK-2989] Error sending message to BlockManagerMaster

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 1.0.2
• Component/s: Block Manager, Spark Core
• Assignee: Unassigned
• Reporter: pengyanhong
• Created: 12/Aug/14 11:58
• Updated: 17/May/20 18:21
• Resolved: 14/Jan/16 03:48

Description

run a simple hive sql Spark App via yarn-cluster, got 3 segments log content via yarn logs --applicationID command line, the detail as below:

• 1st segment is about the Driver & Application Master, everything is fine without error, start time is 16:43:49 and end time is 16:44:08.
• 2nd & 3rd segment is about the Executor, the start time is 16:43:52, then from 16:44:38 encounter many times error as below:

  WARN org.apache.spark.Logging$class.logWarning(Logging.scala:91): Error sending message to BlockManagerMaster in 1 attempts
  java.util.concurrent.TimeoutException: Futures timed out after [30 seconds]
  at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
  at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
  at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
  at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
  at scala.concurrent.Await$.result(package.scala:107)
  at org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:237)
  at org.apache.spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:51)
  at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:113)
  at org.apache.spark.storage.BlockManager$$anonfun$initialize$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(BlockManager.scala:158)
  at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:790)
  at org.apache.spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:158)
  at akka.actor.Scheduler$$anon$9.run(Scheduler.scala:80)
  at akka.actor.LightArrayRevolverScheduler$$anon$3$$anon$2.run(Scheduler.scala:241)
  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  at java.lang.Thread.run(Thread.java:662)
  14/08/12 16:45:31 WARN org.apache.spark.Logging$class.logWarning(Logging.scala:91): Error sending message to BlockManagerMaster in 2 attempts
  ......

confirmed that the date time of 3 nodes is sync.