Details
-
Improvement
-
Status: Open
-
P3
-
Resolution: Unresolved
-
None
-
None
Description
Currently, if we use the `with` statement to run a pipeline, it waits until finish by default with no way to configure it. The only way to run without waiting is by calling `pipeline.run()` explicitly without the `with` statement context. This is not consistent and users now have to be aware of two different ways of calling.
Ideally, we would like to define pipelines with the `with` statement always to have a "single way to do things". Running without waiting is useful, for example, to run a pipeline in Cloud Build, Cloud Run, templates, or other serverless frameworks.
Right now the only way to do it is by changing the code to remove the `with` statement and calling `pipeline.run()` manually, which might not be always possible. With a pipeline option we can just pass that from the command line arguments, or even when testing.
I'm opening this for Python since many Python samples would greatly benefit from this, but I can see this being useful in Java and Go as well.