[SPARK-706] Failures in block manager put leads to task hanging

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 0.6.0, 0.6.1, 0.6.2, 0.7.0
• Component/s: Block Manager, Spark Core
• Assignee: Unassigned
• Reporter: Reynold Xin
• Created: 28/Feb/13 12:55
• Updated: 17/May/20 18:21
• Resolved: 06/Feb/15 21:28

Description

Reported in this thread: https://groups.google.com/forum/?fromgroups=#!topic/shark-users/Q_SiIDzVtZw

The following exception in block manager leaves the block marked as pending.

13/02/26 06:14:56 ERROR executor.Executor: Exception in task ID 39
com.esotericsoftware.kryo.SerializationException: Buffer limit exceeded writing object of type: shark.ColumnarWritable
	at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:492)
	at spark.KryoSerializationStream.writeObject(KryoSerializer.scala:78)
	at spark.serializer.SerializationStream$class.writeAll(Serializer.scala:58)
	at spark.KryoSerializationStream.writeAll(KryoSerializer.scala:73)
	at spark.storage.DiskStore.putValues(DiskStore.scala:63)
	at spark.storage.BlockManager.dropFromMemory(BlockManager.scala:779)
	at spark.storage.MemoryStore.tryToPut(MemoryStore.scala:162)
	at spark.storage.MemoryStore.putValues(MemoryStore.scala:57)
	at spark.storage.BlockManager.put(BlockManager.scala:582)
	at spark.CacheTracker.getOrCompute(CacheTracker.scala:215)
	at spark.RDD.iterator(RDD.scala:159)
	at spark.scheduler.ResultTask.run(ResultTask.scala:18)
	at spark.executor.Executor$TaskRunner.run(Executor.scala:76)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:679)

When the block is read, the task is stuck in BlockInfo.waitForReady().

We should propagate the error back to the master instead of hanging the slave node.

Issue Links

• relates to: SPARK-583 Failures in BlockStore may lead to infinite loops of task failures