Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-5611

One or more consumers in a consumer-group stop consuming after rebalancing

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.2.0
    • Fix Version/s: 0.11.0.1
    • Component/s: None
    • Labels:

      Description

      Scenario:

      • 3 zookeepers, 4 Kafkas. 0.10.2.0, with 0.9.0 compatibility still on (other apps need it but the one mentioned below is already on kafka 0.10.2.0 client).
      • 3 servers running 1 consumer each under the same consumer groupId.
      • Servers seem to be consuming messages happily but then there is a timeout to an external service that causes our app to restart the Kafka Consumer on one of the servers (this is by design). That causes rebalancing of the group and upon restart of one of the Consumers seem to "block".
      • Server 3 is where the problems occur.
      • Problem fixes itself either by restarting one of the 3 servers or cause the group to rebalance again by using the console consumer with the autocommit set to false and using the same group.

      Note:

      • Haven't managed to recreate it at will yet.
      • Mainly happens in production environment, often enough. Hence I do not have any logs with DEBUG/TRACE statements yet.
      • Extracts from log of each app server are attached. Also the log of the kafka that seems to be dealing with the related group and generations.
      • See COMMENT lines in the files for further info.

        Attachments

        1. Server 3
          15 kB
          Panos Skianis
        2. Server 2
          15 kB
          Panos Skianis
        3. Server 1
          15 kB
          Panos Skianis
        4. kka02
          152 kB
          Panos Skianis
        5. bad-server-with-more-logging-1.tar.gz
          34 kB
          Panos Skianis

          Issue Links

            Activity

              People

              • Assignee:
                hachikuji Jason Gustafson
                Reporter:
                pskianis Panos Skianis
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: