[SPARK-11698] Add option to ignore kafka messages that are out of limit rate

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: DStreams
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: L. C. Hsieh
• Created: 12/Nov/15 16:10
• Updated: 21/May/19 04:37
• Resolved: 21/May/19 04:37

Description

With spark.streaming.kafka.maxRatePerPartition, we can control the max rate limit. However, we can not ignore these messages out of limit. These messages will be consumed in next iteration. We have a use case that we need to ignore these messages and process latest messages in next iteration.

In other words, we simply want to consume part of messages in each iteration and ignore remaining messages that are not consumed.

We add an option for this purpose.

Issue Links

• links to: [Github] Pull Request #9665 (viirya)