Currently when launching a job in yarn mode, there will be an added delay of about "spark.yarn.report.interval" seconds before launch. By default this parameter is set to 1 second, however if a user increases this for whatever reason, an unexpected startup delay is introduced.
The "waitForApplication" function called during job submission and launch will eventually call monitorApplication and so sleep for the configured interval time before checking the state of the job and proceeding with job startup.
One solution would be to add an "interval" argument to the monitorApplication function. This added flexibility would allow callers who don't need to wait to set a lower wait interval or none.
Patch to follow soon