[AVRO-747] NettyTransceiver: release semaphores on close so that clients are not blocked.

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.5.0
• Component/s: java
• Assignee: Unassigned
• Reporter: Bruno Dumon
• Created: 27/Jan/11 14:11
• Updated: 12/Aug/11 20:23
• Resolved: 30/Jun/11 19:32

Description

I use Avro RPC with the NettyTransceiver.

When I kill the server, often the client hangs, jstack shows the following:

"pool-6-thread-1" prio=10 tid=0x09fef000 nid=0x3382 waiting on condition [0x76fc7000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xa1df2e40> (a java.util.concurrent.Semaphore$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
        at java.util.concurrent.Semaphore.acquire(Semaphore.java:286)
        at org.apache.avro.ipc.NettyTransceiver$CallFuture.get(NettyTransceiver.java:207)
        at org.apache.avro.ipc.NettyTransceiver.transceive(NettyTransceiver.java:137)
        at org.apache.avro.ipc.Requestor.request(Requestor.java:123)
        - locked <0xa20986c0> (a org.apache.avro.specific.SpecificRequestor)
        at org.apache.avro.specific.SpecificRequestor.invoke(SpecificRequestor.java:52)
...

Not that this matters much, but the client application is written such that it discovers the available servers via ZooKeeper. When a server disappears, it calls close on the corresponding NettyTransceiver.

I have adjusted the NettyTransceiver.close() method to release any remaining semaphores, the same as is done in the exceptionCaught method of the UpstreamHandler. This solves the problem for me.

Alternatively, we could handle channel close events in handleUpstream(), but I'm not sure if Netty automatically reconnects if the server re-appears, in which case this wouldn't be a good idea. OTOH, if the server would never come back, client threads could hang forever?

Patch in attachment, against svn r1064125.