[KAFKA-15392] RestServer starts but does not stop ServletContextHandler

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.7.0
• Component/s: connect
• Assignee: Greg Harris
• Reporter: Greg Harris
• Created: 22/Aug/23 23:24
• Updated: 29/Aug/23 20:03
• Resolved: 29/Aug/23 19:26

Description

Due to the initialization order of the connect RestServer and Herder, the jetty Server is started before the ServletContextHandler instances are installed. This causes jetty to consider them "unmanaged" and thus will not call the start() and stop() lifecycle on our behalf.

RestServer#initializeResources already explicitly calls start() for these unmanaged resources, but there is no accompanying stop() call, so the resources never enter the STOPPED state.

The jetty server has one more operation after stopping: destroy(), which asserts that resources are already stopped. If the jetty server is ever destroyed, this exception will be thrown:

java.lang.IllegalStateException: !STOPPED
at org.eclipse.jetty.server.handler.HandlerWrapper.destroy(HandlerWrapper.java:140)
at o.a.k.connect.runtime.rest.RestServer.stop(RestServer.java:361)

Fortunately, destroy() is currently only called when an error has already occurred, so this IllegalStateException is never thrown on happy-path execution. Instead, if RestServer shutdown encounters an error (such as exceeding the GRACEFUL_SHUTDOWN_TIMEOUT and timing out) the other error will be shadowed by the IllegalStateException.

Rather than only calling destroy() on failure and shadowing the error, destroy() should always be called and it's errors reported separately.

Issue Links

• is related to: KAFKA-15292 Flaky test IdentityReplicationIntegrationTest#testReplicateSourceDefault()
• links to: GitHub Pull Request #14277