[SPARK-3283] Receivers sometimes do not get spread out to multiple nodes

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: DStreams
• Assignee: Tathagata Das
• Reporter: Tathagata Das
• Created: 28/Aug/14 09:14
• Updated: 12/Sep/15 01:04
• Resolved: 12/Sep/15 01:04

Description

The probable reason this happens is because the JobGenerator and JobScheduler start generating jobs with tasks. When the ReceiverTracker submits the task containing receivers, the tasks get assigned according to empty slots, which may be instantaneously available on one node, instead of all the nodes.

The original behavior was that the jobs started only after the receivers are started, thus ensuring that all the slots are free and the receivers are spread evenly across all the nodes.

Issue Links

• is superceded by: SPARK-8882 A New Receiver Scheduling Mechanism to solve unbalanced receivers