Details
-
Bug
-
Status: Triage Needed
-
P3
-
Resolution: Unresolved
-
2.18.0, 2.19.0, 2.20.0, 2.21.0, 2.22.0, 2.23.0, 2.24.0
-
None
-
None
Description
Beam jobs with slow, memory-hungry, or otherwise resource-intensive DoFn implementations perform quite poorly (or even OOM) due to the fact that an UnboundedThreadPoolExecutor is used to spawn workers.
The Python SDK no longer seems to have any methods by which to control concurrent execution of user code. Resource-intensive DoFns can control their own execution by maintaining their own semaphores, but that causes input elements to effectively spool in-memory, with one thread created for every new message. If the input rate of data to a worker exceeds the worker's ability to process those messages, an unbounded number of threads will be spawned to handle incoming work.
Versions of Beam before 2.18 allowed specifying the --worker_threads experimental flag to control concurrency more effectively, but that was removed in November of 2019 by lukecwik@gmail.com (see: BEAM-8151).
One possible solution would be to re-introduce a limit on the size of the _SharedUnboundedThreadPoolExecutor to ensure that we don't create too many threads, but I'm unsure of what kind of backpressure this would create and what effect it may have on the rest of the harness.