Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9935

Kafka not releasing member from Consumer Group

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.4.1
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:
      Linux

      Description

      Hello. I am experiencing an issue where Kafka is not releasing members from a consumer group when the member crashes. The consumer group is then stuck in rebalancing state indefinitely.

      In this consumer group, there is only 1 client. This client has the following related settings:

      auto.commit.interval.ms = 5000
       auto.offset.reset = earliest
       bootstrap.servers = [austgkafka01.hk.eclipseoptions.com:9092]
       check.crcs = true
       client.dns.lookup = default
       client.id = TraderAutomationViewServer_workaround_stuck_rebalance_20200427-0
       connections.max.idle.ms = 540000
       default.api.timeout.ms = 60000
       enable.auto.commit = true
       exclude.internal.topics = true
       fetch.max.bytes = 52428800
       fetch.max.wait.ms = 500
       fetch.min.bytes = 1
       group.id = TraderAutomationViewServer_workaround_stuck_rebalance_20200427
       heartbeat.interval.ms = 3000
       interceptor.classes = []
       internal.leave.group.on.close = true
       isolation.level = read_uncommitted
       key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
       max.partition.fetch.bytes = 1048576
       max.poll.interval.ms = 300000
       max.poll.records = 10000
       metadata.max.age.ms = 300000
       metric.reporters = []
       metrics.num.samples = 2
       metrics.recording.level = INFO
       metrics.sample.window.ms = 30000
       partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
       receive.buffer.bytes = 16777216
       reconnect.backoff.max.ms = 1000
       reconnect.backoff.ms = 50
       request.timeout.ms = 30000
       retry.backoff.ms = 100
       sasl.client.callback.handler.class = null
       sasl.jaas.config = null
       sasl.kerberos.kinit.cmd = /usr/bin/kinit
       sasl.kerberos.min.time.before.relogin = 60000
       sasl.kerberos.service.name = null
       sasl.kerberos.ticket.renew.jitter = 0.05
       sasl.kerberos.ticket.renew.window.factor = 0.8
       sasl.login.callback.handler.class = null
       sasl.login.class = null
       sasl.login.refresh.buffer.seconds = 300
       sasl.login.refresh.min.period.seconds = 60
       sasl.login.refresh.window.factor = 0.8
       sasl.login.refresh.window.jitter = 0.05
       sasl.mechanism = GSSAPI
       security.protocol = PLAINTEXT
       send.buffer.bytes = 131072
       session.timeout.ms = 10000
       ssl.cipher.suites = null
       ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
       ssl.endpoint.identification.algorithm = https
       ssl.key.password = null
       ssl.keymanager.algorithm = SunX509
       ssl.keystore.location = null
       ssl.keystore.password = null
       ssl.keystore.type = JKS
       ssl.protocol = TLS
       ssl.provider = null
       ssl.secure.random.implementation = null
       ssl.trustmanager.algorithm = PKIX
       ssl.truststore.location = null
       ssl.truststore.password = null
       ssl.truststore.type = JKS
       value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
      

      If the client crashes (not a graceful exit from group) the group remains in the following state indefinitely.

      
      Warning: Consumer group 'TraderAutomationViewServer_workaround_stuck_rebalance' is rebalancing.
      
      GROUP                                                 TOPIC             PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
      TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 10         6984061         7839599         855538          -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        8          128459531       143736443       15276912        -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 12         7216495         8106030         889535          -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        6          122921729       137377358       14455629        -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 14         5457618         6171142         713524          -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        4          125647891       140542566       14894675        -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 16         6780884         7557366         776482          -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        2          130906388       146481942       15575554        -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 18         6763721         7611284         847563          -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        16         127231640       142439019       15207379        -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        14         123966927       138503193       14536266        -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        12         132218162       148075484       15857322        -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        10         129898059       145320287       15422228        -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        19         126839746       142013652       15173906        -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 1          6472562         7320125         847563          -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 3          6001010         6680685         679675          -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        0          127150314       142450205       15299891        -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 5          5584947         6222650         637703          -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 7          6394613         7116260         721647          -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 9          6416544         7256574         840030          -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        9          124682226       139534207       14851981        -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 11         5705737         6293935         588198          -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        7          127549142       142739397       15190255        -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 13         6741049         7517531         776482          -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        5          127251434       142368626       15117192        -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 15         6669643         7412866         743223          -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        3          125716835       140706603       14989768        -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 17         4572135         5096785         524650          -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        17         133148873       148911674       15762801        -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 19         4026841         4551491         524650          -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        15         123523794       138414667       14890873        -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        13         126244541       141143311       14898770        -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        11         124746537       139592978       14846441        -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance Dispersion        0          8444970         8444970         0               -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance AutomationMonitor 0          1683809         1849581         165772          -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 0          5453821         6062415         608594          -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance Heartbeat         0          3754819072      3820413525      65594453        -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        18         125249666       140089929       14840263        -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 2          4365938         4890588         524650          -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        1          128224623       143317837       15093214        -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 4          5457939         6150477         692538          -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 6          6703558         7480040         776482          -               -               -
      TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 8          8173671         9127344         953673          -               -               -
      

      The member is not released. Even the process is no longer running

      
      Warning: Consumer group 'TraderAutomationViewServer_workaround_stuck_rebalance' is rebalancing.
      
      GROUP                                                 CONSUMER-ID                                                                                  HOST            CLIENT-ID                                               #PARTITIONS     
      TraderAutomationViewServer_workaround_stuck_rebalance TraderAutomationViewServer_workaround_stuck_rebalance-0-f4e08bf8-e340-44f1-8872-9bcad9ed207a /10.2.16.11     TraderAutomationViewServer_workaround_stuck_rebalance-0 0  
      
      

      The next option here is to create a new group or restart broker.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                skecskes Steve Kecskes
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: