[KAFKA-17223] Retrying the call after encoutering UnsupportedVersionException will cause ConcurrentModificationException

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.9.0
• Component/s: admin, clients
• Assignee: PoAn Yang
• Reporter: Chia-Ping Tsai
• Created: 31/Jul/24 00:22
• Updated: 01/Sep/24 13:26
• Resolved: 08/Aug/24 17:08

Description

[2024-07-31 07:11:03,928] ERROR Uncaught exception in thread 'kafka-admin-client-thread | adminclient-1': (org.apache.kafka.common.utils.KafkaThread:51)
java.util.ConcurrentModificationException
	at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1013)
	at java.base/java.util.ArrayList$Itr.remove(ArrayList.java:981)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.maybeDrainPendingCalls(KafkaAdminClient.java:1207)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1510)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1464)
	at java.base/java.lang.Thread.run(Thread.java:840)

The steps producing above error are shown below.

1. maybeDrainPendingCall[0] encounter error when calling `call.nodeProvider.provide();`[1]
2. `runnable.pendingCalls.add(this)`[2] adds the call back to `pendingCalls`
3. `pendingIter.remove();` tries to remove item from the modified array list.

IMHO, we should add call back to `newCalls` rather than `pendingCalls`. This approach is to revert a part of KAFKA-12432

[0] https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L1206
[1] https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L1219
[2] https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L927
[3] https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L1219

Issue Links

• is related to: KAFKA-15466 Add KIP-919 support for some admin APIs
• is related to: KAFKA-12432 Fix AdminClient timeout handling in the presence of badly behaved brokers
• relates to: KAFKA-17309 Fix flaky testCallFailWithUnsupportedVersionExceptionDoesNotHaveConcurrentModificationException
• links to: GitHub Pull Request #16753