[SPARK-18620] Spark Streaming + Kinesis : Receiver MaxRate is violated

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.2
• Fix Version/s: 2.2.0
• Component/s: DStreams
• Labels: kinesis
• Assignee: Takeshi Yamamuro
• Reporter: david przybill
• Created: 29/Nov/16 09:04
• Updated: 10/May/18 08:32
• Resolved: 09/Dec/16 21:32

Description

I am calling spark-submit passing maxRate, I have a single kinesis receiver, and batches of 1s

spark-submit --conf spark.streaming.receiver.maxRate=10 ....

however a single batch can greatly exceed the stablished maxRate. i.e: Im getting 300 records.

it looks like Kinesis is completely ignoring the spark.streaming.receiver.maxRate configuration.

If you look inside KinesisReceiver.onStart, you see:

val kinesisClientLibConfiguration =
new KinesisClientLibConfiguration(checkpointAppName, streamName, awsCredProvider, workerId)
.withKinesisEndpoint(endpointUrl)
.withInitialPositionInStream(initialPositionInStream)
.withTaskBackoffTimeMillis(500)
.withRegionName(regionName)

This constructor ends up calling another constructor which has a lot of default values for the configuration. One of those values is DEFAULT_MAX_RECORDS which is constantly set to 10,000 records.

Issue Links

• is cloned by: SPARK-23294 Spark Streaming + Rate source + Console Sink : Receiver MaxRate is violated
• links to: [Github] Pull Request #16114 (maropu)