In spark-submit, we currently open a pod watch for any spark submission. If WAIT_FOR_APP_COMPLETION is false, we then immediately ignore the result of the watcher and break out of the watcher.
When submitting spark applications at scale, this is a source of operational pain, since opening the watch relies on opening a websocket, which tends to run into subtle networking issues around negotiating the websocket connection.
I'd like to change this behaviour so that we eagerly check whether we are waiting on app completion, and avoid opening the watch altogether when WAIT_FOR_APP_COMPLETION is false.
Would you accept a contribution for that change, or are there any concerns I've overlooked?