Details
-
Bug
-
Status: Resolved
-
Trivial
-
Resolution: Invalid
-
None
-
None
-
None
-
None
Description
Hi Josh
I am spark user, currently I feel confused about executor registration.
Why CoarsedExecutorBackend register to driver first, and after registerDriver successful, then initial Executor. override def receive: PartialFunction[Any, Unit] = { case RegisteredExecutor(hostname) => logInfo("Successfully registered with driver") executor = new Executor(executorId, hostname, env, userClassPath, isLocal = false)
and it is strange that exit(1) while launchTask with executor=null(due to some errors occurs in Executor initialization).
case LaunchTask(data) => if (executor == null) { logError("Received LaunchTask command but executor was null") System.exit(1)
What concerns was considered to make registerDriver first, Why not register to driver until Executor sth. have been all ready?
like
val (hostname, _) = Utils.extractHostPortFromSparkUrl(hostPort) val executor: Executor = new Executor(executorId, hostname, env, userClassPath, isLocal = false)
Attachments
Issue Links
- duplicates
-
SPARK-13112 CoarsedExecutorBackend register to driver should wait Executor was ready
- Resolved