[PROTON-1000] Connection leak on heartbeat-timeouted connections

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: proton-0.9
• Fix Version/s: proton-0.12.0
• Component/s: python-binding
• Assignee: Gordon Sim
• Reporter: Pavel Moravec
• Created: 21/Sep/15 13:36
• Updated: 16/Feb/16 14:57
• Resolved: 10/Nov/15 10:22

Description

Using gofer/katello-agent that uses BlockingConnection from Proton Reactor with heartbeats set up, if some connection timeouts due to the heartbeats, Proton does not close the TCP connection. That causes TCP connection leak, despite gofer properly called BlockingConnection.close() and forgot any reference to that class instance.

Checking tcpdump, Proton simply ignores the timeouted connections - it does not respond anyhow to the communication partner whatever it sends (in some scenarios it sends some AMQP performative that Proton was assumed to respond, in other scenario the communication peer dropped the TCP connection by sending FIN+ACK packet but Proton didn't send FIN packet back - the only stuff seen in tcpdump is ACKing on TCP layer made by OS, not by Proton). And Proton ignores an attempt of Proton reactor to close the connection/container, raising:

Sep 21 15:02:35 my-capsule goferd: File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 263, in on_transport_closed
Sep 21 15:02:35 my-capsule goferd: raise ConnectionException("Connection %s disconnected" % self.url);
Sep 21 15:02:35 my-capsule goferd: ConnectionException: Connection amqps://satellite.example.com:5647 disconnected

for SSL connections, and raising:

Sep 21 14:56:28 my-capsule goferd: File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 259, in on_transport_tail_closed
Sep 21 14:56:28 my-capsule goferd: self.on_transport_closed(event)
Sep 21 14:56:28 my-capsule goferd: File "/usr/lib64/python2.7/site-packages/proton/utils.py", line 263, in on_transport_closed
Sep 21 14:56:28 my-capsule goferd: raise ConnectionException("Connection %s disconnected" % self.url);
Sep 21 14:56:28 my-capsule goferd: ConnectionException: Connection amqps://satellite.example.com:5647 disconnected

(some difference between SSL and nonSSL could come from the fact that in my case the server part - qdrouterd / Qpid Dispatch Router - sends FIN+ACK packet for nonSSL connection, while it does not send anything for SSL connection and continue for sending empty AMQP frames due to heartbeats enabled forever)

Issue Links

• is required by: PROTON-1010 BlockingConnection leaks sockets after close() is called
• relates to: PROTON-1010 BlockingConnection leaks sockets after close() is called