[SPARK-6443] Support HA in standalone cluster mode

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.4.0
• Component/s: Spark Submit
• Target Version/s: 1.4.0
• Assignee: Tao Wang (Inactive)
• Reporter: Tao Wang (Inactive)
• Created: 21/Mar/15 03:35
• Updated: 09/Jul/15 10:50
• Resolved: 02/May/15 01:38

Description

========== EDIT by Andrew ==========

From a quick survey in the code I can confirm that client mode does support this. This line splits the master URLs by comma and passes these URLs into the AppClient. In standalone cluster mode, there is simply no equivalent logic to even split the master URLs, whether in the old submission gateway (o.a.s.deploy.Client) or in the new one (o.a.s.deploy.rest.StandaloneRestClient).

Thus, this is an unsupported feature, not a bug!

========== Original description from Tao Wang ==========

After digging some codes, I found user could not submit app in standalone cluster mode when HA is enabled. But in client mode it can work.

Haven't try yet. But I will verify this and file a PR to resolve it if the problem exists.

3/23 update:
I started a HA cluster with zk, and tried to submit SparkPi example with command:
./spark-submit --class org.apache.spark.examples.SparkPi --master spark://doggie153:7077,doggie159:7077 --deploy-mode cluster ../lib/spark-examples-1.2.0-hadoop2.4.0.jar

and it failed with error message:
Spark assembly has been built with Hive, including Datanucleus jars on classpath
15/03/23 15:24:45 ERROR actor.OneForOneStrategy: Invalid master URL: spark://doggie153:7077,doggie159:7077
akka.actor.ActorInitializationException: exception during creation
at akka.actor.ActorInitializationException$.apply(Actor.scala:164)
at akka.actor.ActorCell.create(ActorCell.scala:596)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.spark.SparkException: Invalid master URL: spark://doggie153:7077,doggie159:7077
at org.apache.spark.deploy.master.Master$.toAkkaUrl(Master.scala:830)
at org.apache.spark.deploy.ClientActor.preStart(Client.scala:42)
at akka.actor.Actor$class.aroundPreStart(Actor.scala:470)
at org.apache.spark.deploy.ClientActor.aroundPreStart(Client.scala:35)
at akka.actor.ActorCell.create(ActorCell.scala:580)
... 9 more

But in client mode it ended with correct result. So my guess is right. I will fix it in the related PR.

Issue Links

• is duplicated by: SPARK-8941 Standalone cluster worker does not accept multiple masters on launch
• is duplicated by: SPARK-1086 Some corner case during HA master switching?
• is duplicated by: SPARK-5265 Submitting applications on Standalone cluster controlled by Zookeeper forces to know active master
• is duplicated by: SPARK-6801 spark-submit is not able to identify Alive Master in case of multiple master
• links to: [Github] Pull Request #5116 (WangTaoTheTonic)