Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-9835

Enable kafka consumer to subcribe to multiple topics

    Details

    • Type: Wish
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.17.0
    • Fix Version/s: 2.17.1, 2.18.0
    • Component/s: camel-kafka
    • Labels:
      None
    • Environment:

      Any

    • Patch Info:
      Patch Available
    • Estimated Complexity:
      Novice
    • Flags:
      Patch

      Description

      Kafka supports that one consumer is subcriber to multple topics. In their api when you start the consumer you MUST provide an Array of topics.

      If we look in /java/org/apache/camel/component/kafka/KafkaConsumer.java
      Currently this is the piece of code that starts the subscriber:

      LOG.debug("Subscribing {} to topic {}", threadId, topicName);
      consumer.subscribe(Arrays.asList(topicName));
      

      just simply allowing the comma separated topic syntax and replacing one line of code (and the documentation) it would be working.

      LOG.debug("Subscribing {} to topic {}", threadId, topicName);
      consumer.subscribe(Arrays.asList(topicName.split(",")));
      

      Despite the same could be achieved by adding more consumers (rotues) this causes a significant amount of load (because of the commits) to kafka, so this really helps to improve performance.

      the topic has been already marked as mandatory, so that should keep the nullpointer safe.

      @UriParam @Metadata(required = "true")
      private String topic;

      thanks!

        Attachments

          Activity

            People

            • Assignee:
              acosentino Andrea Cosentino
              Reporter:
              jlpedrosa Jose Luis Pedrosa
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 10m
                10m
                Remaining:
                Remaining Estimate - 10m
                10m
                Logged:
                Time Spent - Not Specified
                Not Specified