[KAFKA-7485] Flaky test `DyanamicBrokerReconfigurationTest.testTrustStoreAlter`

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.0
• Assignee: Rajini Sivaram
• Reporter: Jason Gustafson
• Reviewer: Jason Gustafson
• Created: 05/Oct/18 18:12
• Updated: 12/Oct/18 20:37
• Resolved: 12/Oct/18 20:37

Description

09:53:53 
09:53:53 kafka.server.DynamicBrokerReconfigurationTest > testTrustStoreAlter FAILED
09:53:53     org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
09:53:53 
09:53:53         Caused by:
09:53:53         javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 2
09:53:53             at java.base/sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1611)
09:53:53             at java.base/sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:497)
09:53:53             at java.base/sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:745)
09:53:53             at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:680)
09:53:53             at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:626)
09:53:53             at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:474)
09:53:53             at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:274)
09:53:53             at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:126)
09:53:53             at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:532)
09:53:53             at org.apache.kafka.common.network.Selector.poll(Selector.java:467)
09:53:53             at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
09:53:53             at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
09:53:53             at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
09:53:53             at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215)
09:53:53             at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:231)
09:53:53             at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:316)
09:53:53             at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1210)
09:53:53             at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1179)
09:53:53             at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1119)
09:53:53             at kafka.server.DynamicBrokerReconfigurationTest.kafka$server$DynamicBrokerReconfigurationTest$$awaitInitialPositions(DynamicBrokerReconfigurationTest.scala:997)
09:53:53             at kafka.server.DynamicBrokerReconfigurationTest$ConsumerBuilder.build(DynamicBrokerReconfigurationTest.scala:1424)
09:53:53             at kafka.server.DynamicBrokerReconfigurationTest.verifySslProduceConsume$1(DynamicBrokerReconfigurationTest.scala:286)
09:53:53             at kafka.server.DynamicBrokerReconfigurationTest.testTrustStoreAlter(DynamicBrokerReconfigurationTest.scala:311)
09:53:53 
09:53:53             Caused by:
09:53:53             javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 2
09:53:53                 at java.base/sun.security.ssl.HandshakeStateManager.check(HandshakeStateManager.java:398)
09:53:53                 at java.base/sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:215)
09:53:53                 at java.base/sun.security.ssl.Handshaker.processLoop(Handshaker.java:1098)
09:53:53                 at java.base/sun.security.ssl.Handshaker$1.run(Handshaker.java:1031)
09:53:53                 at java.base/sun.security.ssl.Handshaker$1.run(Handshaker.java:1028)
09:53:53                 at java.base/java.security.AccessController.doPrivileged(Native Method)
09:53:53                 at java.base/sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1540)
09:53:53                 at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:399)
09:53:53                 at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:479)
09:53:53                 ... 17 more

Also, it seems we might not be cleaning up the consumer because I see a bunch of subsequent failures due to lingering consumer heartbeat thread.

Issue Links

• links to: GitHub Pull Request #5791