[SPARK-8977] Define the RateEstimator interface, and implement the ReceiverRateController

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: DStreams
• Target Version/s: 1.5.0
• Assignee: Dragos Dascalita Haut
• Reporter: Dragos Dascalita Haut
• Created: 10/Jul/15 15:12
• Updated: 29/Jul/15 23:22
• Resolved: 29/Jul/15 23:22

Description

Full design doc

Implement a rate controller for receiver-based InputDStreams that estimates a maximum rate and sends it to each receiver supervisor.

Issue Links

• blocks: SPARK-8978 Implement the DirectKafkaRateController
• blocks: SPARK-8979 Implement a PIDRateEstimator
• is blocked by: SPARK-8975 Implement a mechanism to send a new rate from the driver to the block generator
• supercedes: SPARK-8834 Throttle DStreams dynamically through back-pressure information
• links to: [Github] Pull Request #7600 (dragos)