[SPARK-29799] Split a kafka partition into multiple KafkaRDD partitions in the kafka external plugin for Spark Streaming

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.1.0
• Component/s: Structured Streaming
• Assignee: Unassigned Assign to me
• Reporter: zengrui
• Created: 08/Nov/19 03:17
• Updated: 13/Apr/20 03:27
• Resolved: 13/Apr/20 03:27

Description

When we use Spark Streaming to consume records from kafka, the generated KafkaRDD‘s partition number is equal to kafka topic's partition number, so we can not use more cpu cores to execute the streaming task except we change the topic's partition number，but we can not increase the topic's partition number infinitely.

Now I think we can split a kafka partition into multiple KafkaRDD partitions, and we can config

it, then we can use more cpu cores to execute the streaming task.

Issue Links

• duplicates: SPARK-23541 Allow Kafka source to read data with greater parallelism than the number of topic-partitions
• links to: GitHub Pull Request #28182