Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
4.4.4
-
None
-
None
Description
We have encountered a few issues with handling an HttpAsyncClient request which is made when the process has too many open file descriptors:
1) A .get() call made on the future returned from the httpclient's .execute() method hangs indefinitely.
2) No call is made to any of the methods on the FutureCallback<HttpResponse> object provided as an argument to the httpclient's .execute() method. I was expecting a call to .failed() to occur.
3) A call to the httpclient's .close() method does not error out but also does not result in all of the IOReactor "I/O Dispatcher" threads being shut down. In normal cases, these threads would get shut down when the .close() call is made.
I've attached a couple of Java source files which demonstrate the problem - AbnormalIOReactor.java and ResponseCallback.java. AbnormalIOReactor.java is based on the example at https://hc.apache.org/httpcomponents-asyncclient-dev/httpasyncclient/examples/org/apache/http/examples/nio/client/AsyncClientHttpExchange.java. Just after the httpclient is started, this class allocates bare sockets repeatedly and holds onto them. This is done to force the application to get into the state where file descriptors have been exhausted. The httpclient.execute() call does not throw an error.
The class repeatedly calls .get() on the future response with a prolonged timeout, breaking out of the loop when the client moves from the state of having .isRunning() be true to .isRunning() being false. Checking for a change in the .isRunning() state is a bit contrived here, ideally something that an API client would not have to do. I noticed from the implementation of the CloseableHttpAsyncClientBase constructor at https://svn.apache.org/repos/asf/httpcomponents/httpasyncclient/tags/4.1.1/httpasyncclient/src/main/java/org/apache/http/impl/nio/client/CloseableHttpAsyncClientBase.java that the reactor thread's run() method will catch an exception at request startup and set the client status to STOPPED and this seemed like the only way that a client could detect that a problem had occurred in this situation.
@Override
public void run() {
try {
final IOEventDispatch ioEventDispatch = new InternalIODispatch(handler);
connmgr.execute(ioEventDispatch);
} catch (final Exception ex) {
log.error("I/O reactor terminated abnormally", ex);
} finally {
status.set(Status.STOPPED);
}
}
The implementation of ResponseCallback.java writes a message to stdout when any of the methods is invoked. In this case, no message is written to stdout even though the request has effectively failed.
When this program is run from the command line like java abnormalioreactor.AbnormalIOReactor, it will hang after the word "Done" appears, which occurs after the .close() method is called on the httpclient. When running jstack from the command line, I see that various "I/O Dispatcher" threads are still running for the client even though I had expected them to have been closed:
"I/O dispatcher 8" #19 prio=5 os_prio=31 tid=0x00007f978e800000 nid=0x6503 runnable [0x0000000120cf6000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000076ba79bf8> (a sun.nio.ch.Util$2)
- locked <0x000000076ba79be8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000076ba79ac8> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:257)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590)
at java.lang.Thread.run(Thread.java:745)
I had been thinking that in this situation it would be better for the httpclient to not be implicitly stopped - so it could be used for subsequent requests if/when the number of open file descriptors has dropped. Also, it would be good for either the httpclient.execute() call to throw an exception right away or at least for calls to .get() on the future returned from the .execute() call to throw an exception and for the .failed() method on FutureCallback<HttpResponse> object to be called.
Note that I ran this application on my MacBook Pro, running OS-X Yosemite. We've seen this problem on other OSes as well, though, including CentOS 7.
I reproduced this problem with a few different versions of HttpAsyncClient - 4.0.2, 4.1.1, custom build from trunk, custom build from the 4.1.x branch.
This isn't a situation that we run into often with our applications - mostly in cases where users haven't tuned their application settings properly to account for the number of file descriptors that a Java process would need to use. It would be nice if the HttpAsyncClient handling could gracefully fail a request when open file descriptors have temporarily been exhausted under load spikes but leave the client in a usable state for future requests.