[SPARK-25233] Give the user the option of specifying a fixed minimum message per partition per batch when using kafka direct API with backpressure

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: DStreams
• Assignee: Reza Safi
• Reporter: Reza Safi
• Created: 24/Aug/18 16:43
• Updated: 30/Aug/18 18:27
• Resolved: 30/Aug/18 18:27

Description

After SPARK-18371, it is guaranteed that there would be at least one message per partition per batch using direct kafka API with backpressure when new messages exist in the topics. It would be better if the user has the option of setting the minimum instead of just a hard coded 1 limit.

Issue Links

• links to: [Github] Pull Request #22223 (rezasafi)