[HTTPCORE-286] IllegalStateException from BaseIOReactor.validate

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.1.1, 4.2-alpha2
• Fix Version/s: 4.1.4, 4.2-beta1
• Component/s: HttpCore NIO
• Environment: Mac OS X 10.6.8; java version "1.6.0_26"; Java(TM) SE Runtime Environment (build 1.6.0_26-b03-384-10M3425); Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02-384, mixed mode)
• Assignee: Unassigned
• Reporter: A. Diamond
• Created: 16/Dec/11 21:30
• Updated: 05/Oct/13 20:09
• Resolved: 18/Dec/11 13:52

Description

We are consistently seeing the following error in the HTTP Async Client, version 4.0-alpha2, running with HTTP Core NIO version 4.1.1:

18:54:38 Thread-79 ERROR org.apache.http.impl.nio.client.DefaultHttpAsyncClient - I/O reactor terminated abnormally
org.apache.http.nio.reactor.IOReactorException: I/O dispatch worker terminated abnormally
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:321)
at org.apache.http.impl.nio.conn.PoolingClientConnectionManager.execute(PoolingClientConnectionManager.java:91)
at org.apache.http.impl.nio.client.AbstractHttpAsyncClient.doExecute(AbstractHttpAsyncClient.java:441)
at org.apache.http.impl.nio.client.AbstractHttpAsyncClient.access$000(AbstractHttpAsyncClient.java:95)
at org.apache.http.impl.nio.client.AbstractHttpAsyncClient$1.run(AbstractHttpAsyncClient.java:462)
Caused by: java.lang.IllegalStateException
at java.util.HashMap$HashIterator.remove(HashMap.java:809)
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:242)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:279)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:542)
at java.lang.Thread.run(Thread.java:662)

This seems to happen when a connection times out. The key line in the stack trace is this one:

at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:242)

This is the code in BaseIOReactor.validate (version 4.1.1) that causes the problem:

232 if ((ops & EventMask.READ) > 0) {
233 try

{ 234 this.eventDispatch.inputReady(session); 235 }

catch (CancelledKeyException ex)

{ 236 it.remove(); 237 queueClosedSession(session); 238 }

catch (RuntimeException ex)

{ 239 handleRuntimeException(ex); 240 }

241 if (!session.hasBufferedInput())

{ 242 it.remove(); 243 }

244 }

You can see the full source for BasIOReactor v4.1.1 here:

http://grepcode.com/file/repo1.maven.org/maven2/org.apache.httpcomponents/httpcore-nio/4.1.1/org/apache/http/impl/nio/reactor/BaseIOReactor.java?av=f

Line 236 calls it.remove(), then the code continues on to call it.remove() again in line 242. Calling it.remove() twice causes java.util.iterator to throw an IllegalStateException. Here's the doc on that:

http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/Iterator.java#Iterator.remove%28%29

There should be a continue statement after line 237, or you should wrap line 242 in a try/catch.

We downloaded the source, wrapped line 242 in a try/catch, and the problem has gone away.

It looks like later versions of BaseIOReactor also have this problem. The code for 4.2-alpha2 at the URL below show the same potential to call it.remove() twice:

http://grepcode.com/file/repo1.maven.org/maven2/org.apache.httpcomponents/httpcore-nio/4.2-alpha2/org/apache/http/impl/nio/reactor/BaseIOReactor.java?av=f

In that code base, the relevant code is in lines 228-240.

Versions between 4.1.1 and 4.2-alpha2 are probably also affected, though I did not take the time to look through all of them.