[SPARK-8834] Throttle DStreams dynamically through back-pressure information

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: DStreams
• Target Version/s: 1.5.0
• Assignee: Unassigned
• Reporter: François Garillot
• Shepherd: Tathagata Das
• Created: 06/Jul/15 09:27
• Updated: 03/Aug/15 22:35
• Resolved: 10/Jul/15 23:30

Description

This aims to have Spark Streaming be more resilient to high-throughput situations through back-pressure signaling & dynamic throttling.

The Design doc can be found there:
https://issues.apache.org/jira/browse/SPARK-8834
An (outdated) PoC implementation exists.

Issue Links

• blocks: SPARK-8835 Provide pluggable Congestion Strategies to deal with Streaming load
• is superceded by: SPARK-8975 Implement a mechanism to send a new rate from the driver to the block generator
• is superceded by: SPARK-8977 Define the RateEstimator interface, and implement the ReceiverRateController
• is superceded by: SPARK-8978 Implement the DirectKafkaRateController
• is superceded by: SPARK-8979 Implement a PIDRateEstimator
• supercedes: SPARK-6691 Abstract and add a dynamic RateLimiter for Spark Streaming