[DIRMINA-779] SSLHandler can re-order data that it reads

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.7, 2.0.0-M1, 2.0.0-M2, 2.0.0-M3, 2.0.0-M4, 2.0.0-M5, 2.0.0-M6, 2.0.0-RC1
• Fix Version/s: 2.0.8
• Component/s: Filter
• Assignee: Unassigned
• Reporter: Jason Resch
• Created: 03/Apr/10 18:54
• Updated: 10/Nov/14 05:15
• Resolved: 11/Sep/14 06:18

Description

The code in question is the flushScheduledEvents() method in SSLHandler.java:

 public void flushScheduledEvents() {
   // Fire events only when no lock is hold for this handler.
   if (Thread.holdsLock(this)) {
       return;
   }

   Event e;
         // We need synchronization here inevitably because filterWrite can be
   // called simultaneously and cause 'bad record MAC' integrity error.
   synchronized (this) {
       while ((e = filterWriteEventQueue.poll()) != null) {
           e.nextFilter.filterWrite(session, (WriteRequest) e.data);
       }
   }

   while ((e = messageReceivedEventQueue.poll()) != null) {
       e.nextFilter.messageReceived(session, e.data);
   }
 }

This method is called both by threads which handle writes, and threads that
handle reads. Therefore, as the comments suggest, multiple threads may go
through this code simultaneously. However, since there is no
synchronization around processing of the messageReceivedEventQueue, it is
possible that the received messages will be sent to the next filter out of
order, should there be more than one message in the queue and a context
switch happen at the wrong time.

The bug would manifest in our application as a failure of our protocol layer
to decode a message, we believe, due to a re-ordering. It only occurred in
environments with a large amount of contention and network traffic and when
using TLS. The fix I have tested was to move the closing brace of the
synchronized block to extend to cover both while loops. I've attached a
patch representing that change. Since making that change we have not
encountered the bug again after about 30 hours of testing and 1.5 TB of
traffic, whereas before the change we could reproduce it after a few
minutes.

Issue Links

• relates to: DIRMINA-995 Deadlock when using SSL and proxy