[THRIFT-5454] Python TProcessPoolServer does not spawn expected number of worker processes

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Component/s: Python - Library
• Labels: Python TProcessPoolServer
• External issue URL: https://github.com/apache/thrift/pull/2438
• Assignee: Yiyang Zhou
• Reporter: Yiyang Zhou
• Created: 13/Aug/21 06:47
• Updated: 15/Feb/22 21:41
• Resolved: 13/Nov/21 23:57

Description

When using Python TProcessPoolServer with more than 1 workers, only one worker process will be successfully spawned and all the other processes will reach a pickling error and exit. As a result, with the default settings, only one worker process is processing requests, whereas 10 workers are expected.

```bash

Traceback (most recent call last):

  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/thrift-0.16.0-py3.9-macosx-10.9-x86_64.egg/thrift/server/TProcessPoolServer.py", line 106, in serve

    w.start()

  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 121, in start

    self._popen = self._Popen(self)

  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 224, in _Popen

    return _default_context.get_context().Process._Popen(process_obj)

  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 284, in _Popen

    return Popen(process_obj)

  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in _init_

    super()._init_(process_obj)

  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py", line 19, in _init_

    self._launch(process_obj)

  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch

    reduction.dump(process_obj, fp)

  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/reduction.py", line 60, in dump

    ForkingPickler(file, protocol).dump(obj)

TypeError: cannot pickle 'weakref' object

```

 

Issue Links

• links to: GitHub Pull Request #2438