Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29799

Split a kafka partition into multiple KafkaRDD partitions in the kafka external plugin for Spark Streaming

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 3.1.0
    • None
    • Structured Streaming
    • None

    Description

      When we use Spark Streaming to consume records from kafka, the generated KafkaRDD‘s partition number is equal to kafka topic's partition number, so we can not use more cpu cores to execute the streaming task except we change the topic's partition number,but we can not increase the topic's partition number infinitely.

      Now I think we can split a kafka partition into multiple KafkaRDD partitions, and we can config

      it, then we can use more cpu cores to execute the streaming task.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              zengrui zengrui
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: