Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-10413

rebalancing leads to unevenly balanced connectors

    XMLWordPrintableJSON

Details

    Description

      GHi,

      With CP 5.5, running kafka connect s3 sink on EC2 whith autoscaling enabled, if a connect instance disappear, or a new one appear, we're seeing unbalanced consumption, much like mentionned in this post:

      https://stackoverflow.com/questions/58644622/incremental-cooperative-rebalancing-leads-to-unevenly-balanced-connectors

      This usually leads to one kafka connect instance taking most of the load and consumption not being able to keep on.
      Currently, we're "fixing" this by deleting the connector and re-creating it, but this is far from ideal.

      Any suggestion on what we could do to mitigate this ?

      Attachments

        Issue Links

          Activity

            People

              ramkrish1489 rameshkrishnan muthusamy
              yazgoo yazgoo
              Votes:
              5 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated: