[KAFKA-9935] Kafka not releasing member from Consumer Group

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.4.1
• Environment: Linux
• Assignee: Unassigned
• Reporter: Steve Kecskes
• Created: 29/Apr/20 07:36
• Updated: 09/Aug/20 22:06
• Resolved: 09/Aug/20 22:06

Description

Hello. I am experiencing an issue where Kafka is not releasing members from a consumer group when the member crashes. The consumer group is then stuck in rebalancing state indefinitely.

In this consumer group, there is only 1 client. This client has the following related settings:

auto.commit.interval.ms = 5000
 auto.offset.reset = earliest
 bootstrap.servers = [austgkafka01.hk.eclipseoptions.com:9092]
 check.crcs = true
 client.dns.lookup = default
 client.id = TraderAutomationViewServer_workaround_stuck_rebalance_20200427-0
 connections.max.idle.ms = 540000
 default.api.timeout.ms = 60000
 enable.auto.commit = true
 exclude.internal.topics = true
 fetch.max.bytes = 52428800
 fetch.max.wait.ms = 500
 fetch.min.bytes = 1
 group.id = TraderAutomationViewServer_workaround_stuck_rebalance_20200427
 heartbeat.interval.ms = 3000
 interceptor.classes = []
 internal.leave.group.on.close = true
 isolation.level = read_uncommitted
 key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
 max.partition.fetch.bytes = 1048576
 max.poll.interval.ms = 300000
 max.poll.records = 10000
 metadata.max.age.ms = 300000
 metric.reporters = []
 metrics.num.samples = 2
 metrics.recording.level = INFO
 metrics.sample.window.ms = 30000
 partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
 receive.buffer.bytes = 16777216
 reconnect.backoff.max.ms = 1000
 reconnect.backoff.ms = 50
 request.timeout.ms = 30000
 retry.backoff.ms = 100
 sasl.client.callback.handler.class = null
 sasl.jaas.config = null
 sasl.kerberos.kinit.cmd = /usr/bin/kinit
 sasl.kerberos.min.time.before.relogin = 60000
 sasl.kerberos.service.name = null
 sasl.kerberos.ticket.renew.jitter = 0.05
 sasl.kerberos.ticket.renew.window.factor = 0.8
 sasl.login.callback.handler.class = null
 sasl.login.class = null
 sasl.login.refresh.buffer.seconds = 300
 sasl.login.refresh.min.period.seconds = 60
 sasl.login.refresh.window.factor = 0.8
 sasl.login.refresh.window.jitter = 0.05
 sasl.mechanism = GSSAPI
 security.protocol = PLAINTEXT
 send.buffer.bytes = 131072
 session.timeout.ms = 10000
 ssl.cipher.suites = null
 ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
 ssl.endpoint.identification.algorithm = https
 ssl.key.password = null
 ssl.keymanager.algorithm = SunX509
 ssl.keystore.location = null
 ssl.keystore.password = null
 ssl.keystore.type = JKS
 ssl.protocol = TLS
 ssl.provider = null
 ssl.secure.random.implementation = null
 ssl.trustmanager.algorithm = PKIX
 ssl.truststore.location = null
 ssl.truststore.password = null
 ssl.truststore.type = JKS
 value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer

If the client crashes (not a graceful exit from group) the group remains in the following state indefinitely.

Warning: Consumer group 'TraderAutomationViewServer_workaround_stuck_rebalance' is rebalancing.

GROUP                                                 TOPIC             PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 10         6984061         7839599         855538          -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        8          128459531       143736443       15276912        -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 12         7216495         8106030         889535          -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        6          122921729       137377358       14455629        -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 14         5457618         6171142         713524          -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        4          125647891       140542566       14894675        -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 16         6780884         7557366         776482          -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        2          130906388       146481942       15575554        -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 18         6763721         7611284         847563          -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        16         127231640       142439019       15207379        -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        14         123966927       138503193       14536266        -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        12         132218162       148075484       15857322        -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        10         129898059       145320287       15422228        -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        19         126839746       142013652       15173906        -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 1          6472562         7320125         847563          -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 3          6001010         6680685         679675          -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        0          127150314       142450205       15299891        -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 5          5584947         6222650         637703          -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 7          6394613         7116260         721647          -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 9          6416544         7256574         840030          -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        9          124682226       139534207       14851981        -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 11         5705737         6293935         588198          -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        7          127549142       142739397       15190255        -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 13         6741049         7517531         776482          -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        5          127251434       142368626       15117192        -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 15         6669643         7412866         743223          -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        3          125716835       140706603       14989768        -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 17         4572135         5096785         524650          -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        17         133148873       148911674       15762801        -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 19         4026841         4551491         524650          -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        15         123523794       138414667       14890873        -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        13         126244541       141143311       14898770        -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        11         124746537       139592978       14846441        -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance Dispersion        0          8444970         8444970         0               -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance AutomationMonitor 0          1683809         1849581         165772          -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 0          5453821         6062415         608594          -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance Heartbeat         0          3754819072      3820413525      65594453        -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        18         125249666       140089929       14840263        -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 2          4365938         4890588         524650          -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance VolMetrics        1          128224623       143317837       15093214        -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 4          5457939         6150477         692538          -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 6          6703558         7480040         776482          -               -               -
TraderAutomationViewServer_workaround_stuck_rebalance EventAdjustedVols 8          8173671         9127344         953673          -               -               -

The member is not released. Even the process is no longer running

Warning: Consumer group 'TraderAutomationViewServer_workaround_stuck_rebalance' is rebalancing.

GROUP                                                 CONSUMER-ID                                                                                  HOST            CLIENT-ID                                               #PARTITIONS     
TraderAutomationViewServer_workaround_stuck_rebalance TraderAutomationViewServer_workaround_stuck_rebalance-0-f4e08bf8-e340-44f1-8872-9bcad9ed207a /10.2.16.11     TraderAutomationViewServer_workaround_stuck_rebalance-0 0  

The next option here is to create a new group or restart broker.

Issue Links

• duplicates: KAFKA-9752 Consumer rebalance can be stuck after new member timeout with old JoinGroup version
• is cloned by: KAFKA-10105 Regression in group coordinator dealing with flaky clients joining while leaving