Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16019

Eliminate unexpected delay during spark on yarn job launch

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.3.0
    • Component/s: YARN
    • Labels:
      None

      Description

      Currently when launching a job in yarn mode, there will be an added delay of about "spark.yarn.report.interval" seconds before launch. By default this parameter is set to 1 second, however if a user increases this for whatever reason, an unexpected startup delay is introduced.

      The "waitForApplication" function called during job submission and launch will eventually call monitorApplication and so sleep for the configured interval time before checking the state of the job and proceeding with job startup.

      One solution would be to add an "interval" argument to the monitorApplication function. This added flexibility would allow callers who don't need to wait to set a lower wait interval or none.

      Patch to follow soon

        Attachments

          Activity

            People

            • Assignee:
              vanzin Marcelo Vanzin
              Reporter:
              olasoji Olasoji
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: