Have a --wait_until_finish=[True|False] pipeline option

[damccorm]

Currently, if we use the with statement to run a pipeline, it waits until finish by default with no way to configure it. The only way to run without waiting is by calling pipeline.run() explicitly without the with statement context. This is not consistent and users now have to be aware of two different ways of calling.

Ideally, we would like to define pipelines with the with statement always to have a "single way to do things". Running without waiting is useful, for example, to run a pipeline in Cloud Build, Cloud Run, templates, or other serverless frameworks.

Right now the only way to do it is by changing the code to remove the with statement and calling pipeline.run() manually, which might not be always possible. With a pipeline option we can just pass that from the command line arguments, or even when testing.

I'm opening this for Python since many Python samples would greatly benefit from this, but I can see this being useful in Java and Go as well.

Imported from Jira BEAM-12455. Original Jira may contain additional context.
Reported by: dcavazos.

[liferoad]

.take-issue