This aims to have Spark Streaming be more resilient to high-throughput situations through back-pressure signaling & dynamic throttling.
The Design doc can be found there:
An (outdated) PoC implementation exists.
Provide pluggable Congestion Strategies to deal with Streaming load
Implement a mechanism to send a new rate from the driver to the block generator
Define the RateEstimator interface, and implement the ReceiverRateController
Implement the DirectKafkaRateController
Implement a PIDRateEstimator
Abstract and add a dynamic RateLimiter for Spark Streaming