Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35174

Avoid opening watch when waitAppCompletion is false

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.1.1
    • Fix Version/s: 3.3.0
    • Component/s: Kubernetes
    • Labels:
      None

      Description

      In spark-submit, we currently open a pod watch for any spark submission. If WAIT_FOR_APP_COMPLETION is false, we then immediately ignore the result of the watcher and break out of the watcher.

      When submitting spark applications at scale, this is a source of operational pain, since opening the watch relies on opening a websocket, which tends to run into subtle networking issues around negotiating the websocket connection.

      I'd like to change this behaviour so that we eagerly check whether we are waiting on app completion, and avoid opening the watch altogether when WAIT_FOR_APP_COMPLETION is false.

      Would you accept a contribution for that change, or are there any concerns I've overlooked?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jlafleche Jonathan Lafleche
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: