[THRIFT-2918] Race condition in Python TProcessPoolServer test

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.3
• Component/s: Python - Library

Environment

openSUSE 13.2
Python 2.7.8 (default, Sep 30 2014, 15:34:38) [GCC] on linux2

• Assignee: Jens Geyer
• Reporter: Jens Geyer
• Created: 28/Dec/14 23:49
• Updated: 17/Sep/15 08:27
• Resolved: 25/Jul/15 13:45

Description

make check gets stuck very reproducible in my VM at Test run #218:

Test run #217:  (includes gen-py-default) Server=TProcessPoolServer,  Proto=accel,  zlib=False,  SSL=False
Testing server TProcessPoolServer: /usr/bin/python ./TestServer.py --genpydir=gen-py-default --protocol=accel --port=9090 TProcessPoolServer
Testing client: /usr/bin/python ./TestClient.py --genpydir=gen-py-default --protocol=accel --port=9090 --transport=buffered
...testException(Safe)
testException(Xception)
testException(throw_undeclared)
...............
----------------------------------------------------------------------
Ran 18 tests in 0.563s

OK
Giving TProcessPoolServer (proto=accel,zlib=False,ssl=False) an extra 3 seconds for childprocesses to terminate via alarm
Terminating worker: <Process(Process-1, started daemon)>
Terminating worker: <Process(Process-2, started daemon)>
Terminating worker: <Process(Process-3, started daemon)>
Terminating worker: <Process(Process-4, started daemon)>
Terminating worker: <Process(Process-5, started daemon)>
Requesting server to stop()
OK: Finished (includes gen-py-default)  TProcessPoolServer / accel proto / zlib=False / SSL=False.   217 combinations tested.

Test run #218:  (includes gen-py-default) Server=TProcessPoolServer,  Proto=accel,  zlib=False,  SSL=True
Testing server TProcessPoolServer: /usr/bin/python ./TestServer.py --genpydir=gen-py-default --protocol=accel --port=9090 --ssl TProcessPoolServer
Testing client: /usr/bin/python ./TestClient.py --genpydir=gen-py-default --protocol=accel --port=9090 --ssl --transport=buffered
...testException(Safe)
testException(Xception)
testException(throw_undeclared)
..........Terminating worker: <Process(Process-1, started daemon)>
Terminating worker: <Process(Process-2, started daemon)>
Terminating worker: <Process(Process-3, started daemon)>
Terminating worker: <Process(Process-4, started daemon)>
Terminating worker: <Process(Process-5, started daemon)>
Requesting server to stop()

After fiddling a bit around with it I got it to work by increasing the alarm() timeout from 2 seconds to 4 seconds.

I'm not a Python expert, but the code looks somewhat interesting to me:

• The server code starts the workers, but some piece of code outside of the server is responsible for terminating them. Is that really idiomatic in Python or just bad design?
• The Condition() object used in TProcessPoolsServer.py should probably be replaced by an Event() object. Especially, as Condition.wait() seems to have it's own perils and Event is much easier to use.
• Calling Condition.aquire() without a matching release() within a while True: loop looks also not very convincing to me. AFAIK the second call to aquire() will block, if that ever happens (it did not in my tests).

The bad news is, that neither of the proposed changes above had any effect on the race conditions, except increasing the timeout - but that is merely a workaround, not a solution.