[HTTPCORE-654] Unhandled Exception Cause I/O Reactor to Shutdown

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 4.4.13
• Component/s: HttpCore NIO
• Assignee: Unassigned
• Reporter: Danmeng Tu
• Created: 10/Nov/20 10:47
• Updated: 10/Nov/20 20:41
• Resolved: 10/Nov/20 14:34

Description

Hi team,

Recently we encountered an unhandled exception that triggered I/O reactor to shutdown:

ERROR 2020-11-10 18:20:50 [pool-1-thread-1] o.a.h.i.n.c.InternalHttpAsyncClient - I/O reactor terminated abnormally
java.lang.IllegalStateException: Response is closed
	at io.vertx.core.http.impl.HttpServerResponseImpl.checkValid(HttpServerResponseImpl.java:617) ~[vertx-core-3.7.1.jar!/:3.7.1]
	at io.vertx.core.http.impl.HttpServerResponseImpl.putHeader(HttpServerResponseImpl.java:157) ~[vertx-core-3.7.1.jar!/:3.7.1]
	at io.vertx.core.http.impl.HttpServerResponseImpl.putHeader(HttpServerResponseImpl.java:54) ~[vertx-core-3.7.1.jar!/:3.7.1]
	at com.coupang.listing.listingimageservice.endpoint.ListingImageController.lambda$null$1(ListingImageController.java:87) ~[classes!/:na]
	at io.vertx.core.impl.CompositeFutureImpl.lambda$all$1(CompositeFutureImpl.java:49) ~[vertx-core-3.7.1.jar!/:3.7.1]
	at io.vertx.core.impl.FutureImpl.tryComplete(FutureImpl.java:125) ~[vertx-core-3.7.1.jar!/:3.7.1]
	at io.vertx.core.impl.FutureImpl.complete(FutureImpl.java:86) ~[vertx-core-3.7.1.jar!/:3.7.1]
	at com.coupang.listing.listingimageservice.component.ImageProcessor.handleDownloadResult(ImageProcessor.java:184) ~[classes!/:na]
	at com.coupang.listing.listingimageservice.component.ImageProcessor.lambda$process$2(ImageProcessor.java:135) ~[classes!/:na]
	at io.vertx.core.Future.lambda$compose$0(Future.java:227) ~[vertx-core-3.7.1.jar!/:3.7.1]
	at io.vertx.core.impl.FutureImpl.tryComplete(FutureImpl.java:125) ~[vertx-core-3.7.1.jar!/:3.7.1]
	at io.vertx.core.impl.FutureImpl.complete(FutureImpl.java:86) ~[vertx-core-3.7.1.jar!/:3.7.1]
	at com.coupang.listing.listingimageservice.component.imageInfo.downloader.ResourceDownloadFacade.lambda$handler$0(ResourceDownloadFacade.java:51) ~[classes!/:na]
	at io.vertx.core.impl.FutureImpl.tryFail(FutureImpl.java:170) ~[vertx-core-3.7.1.jar!/:3.7.1]
	at io.vertx.core.impl.FutureImpl.fail(FutureImpl.java:100) ~[vertx-core-3.7.1.jar!/:3.7.1]
	at com.coupang.listing.listingimageservice.component.imageInfo.downloader.downloaders.AbstractHttpClientDownloader$1.failed(AbstractHttpClientDownloader.java:122) ~[classes!/:na]
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:137) ~[httpcore-4.4.13.jar!/:4.4.13]
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:101) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
	at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:426) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
	at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.connectionRequestFailed(AbstractClientExchangeHandler.java:348) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
	at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.access$100(AbstractClientExchangeHandler.java:62) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
	at org.apache.http.impl.nio.client.AbstractClientExchangeHandler$1.failed(AbstractClientExchangeHandler.java:392) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:137) ~[httpcore-4.4.13.jar!/:4.4.13]
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$1.failed(PoolingNHttpClientConnectionManager.java:316) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:137) ~[httpcore-4.4.13.jar!/:4.4.13]
	at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:169) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
	at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:632) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
	at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:898) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
	at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:198) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:213) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:158) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]

another case

ERROR 2020-11-10 17:38:38 [pool-1-thread-1] o.a.h.i.n.c.InternalHttpAsyncClient - I/O reactor terminated abnormally
org.apache.http.nio.reactor.IOReactorException: I/O dispatch worker terminated abnormally
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:359) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Caused by: java.lang.IllegalStateException: Result is already complete: succeeded
	at io.vertx.core.impl.FutureImpl.complete(FutureImpl.java:87) ~[vertx-core-3.7.1.jar!/:3.7.1]
	at com.coupang.listing.listingimageservice.component.ImageProcessor.lambda$null$0(ImageProcessor.java:117) ~[classes!/:na]
	at io.vertx.core.impl.FutureImpl.tryComplete(FutureImpl.java:125) ~[vertx-core-3.7.1.jar!/:3.7.1]
	at io.vertx.core.impl.FutureImpl.complete(FutureImpl.java:86) ~[vertx-core-3.7.1.jar!/:3.7.1]
	at com.coupang.listing.listingimageservice.component.imageInfo.downloader.ResourceDownloadFacade.lambda$handler$0(ResourceDownloadFacade.java:51) ~[classes!/:na]
	at io.vertx.core.impl.FutureImpl.tryFail(FutureImpl.java:170) ~[vertx-core-3.7.1.jar!/:3.7.1]
	at io.vertx.core.impl.FutureImpl.fail(FutureImpl.java:100) ~[vertx-core-3.7.1.jar!/:3.7.1]
	at com.coupang.listing.listingimageservice.component.imageInfo.downloader.downloaders.AbstractHttpClientDownloader$1.failed(AbstractHttpClientDownloader.java:122) ~[classes!/:na]
	at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:137) ~[httpcore-4.4.13.jar!/:4.4.13]
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:101) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
	at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:426) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.exception(HttpAsyncRequestExecutor.java:163) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:276) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
	... 1 common frames omitted

code
AbstractHttpClientDownloader.java

Let me know if additional info is needed please, thank you.