[SPARK-23294] Spark Streaming + Rate source + Console Sink : Receiver MaxRate is violated

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 2.2.1
• Component/s: Structured Streaming
• Labels: ConsoleSink RateSource backpressure maxRate
• Assignee: Takeshi Yamamuro
• Reporter: Ravinder Matte
• Created: 01/Feb/18 06:13
• Updated: 01/Feb/18 22:35
• Resolved: 01/Feb/18 22:35

Description

Using following configs while building SparkSession

("spark.streaming.backpressure.enabled", "true")
("spark.streaming.receiver.maxRate", "100")
("spark.streaming.backpressure.initialRate", "100")

 

Source: Rate Source with following options.

rowsPerSecond=100000

Sink: Console Sink. 

 

I am expecting the process rate to limit at 100 Rows per Second, but maxRate is ignored and streaming job is processing at 100000 rate.

 

Issue Links

• is a clone of: SPARK-18620 Spark Streaming + Kinesis : Receiver MaxRate is violated