[SPARK-4986] Graceful shutdown for Spark Streaming does not work in Standalone cluster mode

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.2.2, 1.3.0
• Component/s: DStreams
• Assignee: Jesper Lundgren
• Reporter: Jesper Lundgren
• Created: 29/Dec/14 07:28
• Updated: 30/Apr/15 15:57
• Resolved: 03/Feb/15 23:23

Description

When using the graceful stop API of Spark Streaming in Spark Standalone cluster the stop signal never reaches the receivers. I have tested this with Spark 1.2 and Kafka receivers.

ReceiverTracker will send StopReceiver message to ReceiverSupervisorImpl.
In local mode ReceiverSupervisorImpl receives this message but in Standalone cluster mode the message seems to be lost.

(I have modified the code to send my own string message as a stop signal from ReceiverTracker to ReceiverSupervisorImpl and it works as a workaround.)

Issue Links

• relates to: SPARK-4545 If first Spark Streaming batch fails, it waits 10x batch duration before stopping
• links to: [Github] Pull Request #3868 (cleaton)
• links to: [Github] Pull Request #4338 (cleaton)