[SPARK-36545] sparkstreaming input rate exceed spark.streaming.kafka.maxRatePerPartition

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.4.5
• Component/s: DStreams, Spark Submit, YARN
• Assignee: Unassigned
• Reporter: handong
• Created: 19/Aug/21 11:25
• Updated: 12/Dec/22 18:10
• Resolved: 23/Aug/21 01:21

Description

 

i have a spark streaming application with kafka . 
Here are the parameters:

kafka partition = 500

batch time = 60 

--conf spark.streaming.backpressure.enabled=true

--conf spark.streaming.kafka.maxRatePerPartition=2500

input size= 500 * 120 * 2500 = 75,000,000

however input size become 160000000 after some batch 

who can tell me reason