[BEAM-10200] Improve memory profiling for users of Portable Beam Python

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: sdk-py-harness
• Labels: starter
• Assignee: Unassigned
• Reporter: Valentyn Tymofieiev
• Created: 05/Jun/20 01:11
• Updated: 04/Jun/22 16:29

Description

We have a Profiler[1] that is integrated with SDK worker[1a], however it only saves CPU metrics [1b].
We have a MemoryReporter util[2] which can log heap dumps, however it is not documented on Beam Website and does not respect the --profile_memory and --profile_location options[3]. The profile_memory flag currently works only for Dataflow Runner users who run non-portable batch pipelines; profiles are saved only if memory usage between samples exceeds 1000M.

We should improve memory profiling experience for Portable Python users and consider making a guide on how users can investigate OOMing pipelines on Beam website.

[1] https://github.com/apache/beam/blob/095589c28f5c427bf99fc0330af91c859bb2ad6b/sdks/python/apache_beam/utils/profiler.py#L46
[1a] https://github.com/apache/beam/blob/095589c28f5c427bf99fc0330af91c859bb2ad6b/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L157
[1b] https://github.com/apache/beam/blob/095589c28f5c427bf99fc0330af91c859bb2ad6b/sdks/python/apache_beam/utils/profiler.py#L112
[2] https://github.com/apache/beam/blob/095589c28f5c427bf99fc0330af91c859bb2ad6b/sdks/python/apache_beam/utils/profiler.py#L124
[3] https://github.com/apache/beam/blob/095589c28f5c427bf99fc0330af91c859bb2ad6b/sdks/python/apache_beam/options/pipeline_options.py#L846

Issue Links

• links to: GitHub Pull Request #12537
• links to: GitHub Pull Request #12562