[HTTPCORE-695] Unhandled CancelledKeyException during processPendingInterestOps leads to a shutdown of the underlying IOReactor

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.4.10
• Fix Version/s: 4.4.15
• Component/s: HttpCore NIO
• Environment: Oracle Java 11.0.9.0.2 on Linux 3.10.0-1127.19.1.el7.x86_64
• Assignee: Unassigned
• Reporter: Kurtcebe Eroglu
• Created: 04/Nov/21 17:58
• Updated: 05/Nov/21 20:27
• Resolved: 04/Nov/21 20:10

Description

An unhandled java.nio.channels.CancelledKeyException in AbstractIOReactor#processPendingInterestOps() leads to a shutdown of the underlying IOReactor. This is very similar to HTTPCORE-336.

This happens regularly under non-reproducible circumstances, producing the following stack trace in the logs;

2021-10-12 00:22:19,326-0400 httpclient-io:thread-1 ERROR anonymous     [o.a.h.i.nio.client.InternalHttpAsyncClient] I/O reactor terminated abnormally
org.apache.http.nio.reactor.IOReactorException: I/O dispatch worker terminated abnormally
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:356)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.nio.channels.CancelledKeyException
	at java.base/sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:71)
	at java.base/sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:96)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processPendingInterestOps(AbstractIOReactor.java:459)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:298)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)

The problem happens at AbstractIOReactor#processPendingInterestOps(). Even if we check key.isValid() in the loop before processing, apparently we may still observe a CancelledKeyException during key.interestOps(eventMask). Following patch may help this condition, following from similar check at method processClosedSessions() ;

Index: httpcore-nio/src/main/java/org/apache/http/impl/nio/reactor/AbstractIOReactor.java
===================================================================
--- httpcore-nio/src/main/java/org/apache/http/impl/nio/reactor/AbstractIOReactor.java	(revision 975b373b6e0431ba8f58abe004039f925703a808)
+++ httpcore-nio/src/main/java/org/apache/http/impl/nio/reactor/AbstractIOReactor.java	(date 1636046242908)
@@ -456,7 +456,11 @@
             final SelectionKey key = entry.getSelectionKey();
             final int eventMask = entry.getEventMask();
             if (key.isValid()) {
-                key.interestOps(eventMask);
+                try {
+                    key.interestOps(eventMask);
+                } catch (CancelledKeyException ex) {
+                    // ignore and move on
+                }
             }
         }
     }