[THRIFT-2883] TTwisted.py, during ConnectionLost processing: exceptions.RuntimeError: dictionary changed size during iteration

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.2
• Fix Version/s: 0.10.0
• Component/s: Python - Library
• Labels: easyfix patch
• Patch Info: Patch Available
• Assignee: Unassigned
• Reporter: Robert P. Thille
• Created: 10/Dec/14 00:43
• Updated: 06/Jan/16 02:06
• Resolved: 23/Dec/15 17:44

Description

If any of the errbacks called by the connectionLost routine attempt to make new requests over thrift (retries), the requests dictionary will change size and Python with throw the RuntimeError exception and the clients will not be properly notified of the connectionLost state change.
2014-12-06 02:22:16,808 ERROR twisted Unhandled Error
Traceback (most recent call last):
File "/usr/lib/pymodules/python2.7/companyutils/service/_init_.py", line 292, in main
self.reactor.run()
File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1169, in run
self.mainLoop()
File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1181, in mainLoop
self.doIteration(t)
File "/usr/lib/python2.7/dist-packages/twisted/internet/epollreactor.py", line 379, in doPoll
log.callWithLogger(selectable, _drdw, selectable, fd, event)
— <exception caught here> —
File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 84, in callWithLogger
return callWithContext(

{"system": lp}

, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 69, in callWithContext
return context.call(

{ILogContext: newCtx}

, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
return func(args,*kw)
File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 627, in _doReadOrWrite
self._disconnectSelectable(selectable, why, inRead)
File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 258, in _disconnectSelectable
selectable.readConnectionLost(f)
File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 267, in readConnectionLost
self.connectionLost(reason)
File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 473, in connectionLost
self._commonConnection.connectionLost(self, reason)
File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 287, in connectionLost
protocol.connectionLost(reason)
File "/usr/lib/pymodules/python2.7/telephus/protocol.py", line 63, in connectionLost
self._parent_protocol.connectionLost(self, reason)
File "/usr/lib/pymodules/python2.7/thrift/transport/TTwisted.py", line 84, in connectionLost
for k, v in self.client._reqs.iteritems():
exceptions.RuntimeError: dictionary changed size during iteration