[SPARK-14455] ReceiverTracker#allocatedExecutors throw NPE for receiver-less streaming application

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: DStreams
• Target Version/s: 2.0.0
• Assignee: Saisai Shao
• Reporter: Saisai Shao
• Created: 07/Apr/16 08:24
• Updated: 10/Apr/16 06:34
• Resolved: 10/Apr/16 06:34

Description

When testing streaming dynamic allocation with direct Kafka, streaming ExecutorAllocationManager throws NPE which is due to ReceiverTracker not start, it will happen when running receiver-less streaming application like direct Kafka. Here is the exception log:

Exception in thread "RecurringTimer - streaming-executor-allocation-manager" java.lang.NullPointerException
	at org.apache.spark.streaming.scheduler.ReceiverTracker.allocatedExecutors(ReceiverTracker.scala:244)
	at org.apache.spark.streaming.scheduler.ExecutorAllocationManager.killExecutor(ExecutorAllocationManager.scala:124)
	at org.apache.spark.streaming.scheduler.ExecutorAllocationManager.org$apache$spark$streaming$scheduler$ExecutorAllocationManager$$manageAllocation(ExecutorAllocationManager.scala:100)
	at org.apache.spark.streaming.scheduler.ExecutorAllocationManager$$anonfun$1.apply$mcVJ$sp(ExecutorAllocationManager.scala:66)
	at org.apache.spark.streaming.util.RecurringTimer.triggerActionForNextInterval(RecurringTimer.scala:94)
	at org.apache.spark.streaming.util.RecurringTimer.org$apache$spark$streaming$util$RecurringTimer$$loop(RecurringTimer.scala:106)
	at org.apache.spark.streaming.util.RecurringTimer$$anon$1.run(RecurringTimer.scala:29)

Issue Links

• links to: [Github] Pull Request #12236 (jerryshao)