[BEAM-7790] Make debugging subprocess workers easier

• Type: Improvement
• Status: Resolved
• Priority: P3
• Resolution: Duplicate
• Fix Version/s: Not applicable
• Component/s: sdk-py-harness
• Labels: stale-assigned
• Assignee: Kyle Weaver
• Reporter: Kyle Weaver
• Created: 20/Jul/19 21:00
• Updated: 02/Jun/20 15:03
• Resolved: 02/Jun/20 15:03

Description

[ ] The output of the SDK workers is currently invisible due to the output and logging setup.

[ ] The dockerized version of the Python SDK worker sets up an HTTP server to let the user view stack traces for all of the worker's threads [1]. It would be useful if this was available for other execution modes as well.

[x] BEAM-7676 Make the above items more usable with multiple subprocesses by identifying them with worker ids.

 

[1] https://github.com/apache/beam/blob/9f4ce1c6fc2fb195e218783a6e9ce6104ddb4d1e/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L46-L89

Issue Links

• duplicates: BEAM-8278 Get output from PROCESS environment
• is blocked by: BEAM-7676 All SDK workers have worker_id="1"
• is related to: BEAM-8015 Get logs for SDK worker Docker containers