[BEAM-3072] Improve error handling at staging time for DataflowRunner

• Type: Bug
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Fix Version/s: 2.12.0
• Component/s: sdk-py-core
• Labels: starter
• Assignee: niklas Hansson
• Reporter: Ahmet Altay
• Created: 17/Oct/17 21:52
• Updated: 16/May/20 13:20
• Resolved: 18/Apr/19 17:11

Description

dependency.py calls out to external process to collect dependencies:

https://github.com/apache/beam/blob/de7cc05cc67d1aa6331cddc17c2e02ed0efbe37d/sdks/python/apache_beam/runners/dataflow/internal/dependency.py#L263

If these calls fails, the error is not clear. The error only tells what failed but does not show the actual error message, and is not helpful for users.

As a general fix processes.py should have general better output collection from failed processes.

Issue Links

• links to: GitHub Pull Request #8158
• links to: GitHub Pull Request #8337