[LIVY-685] Running Livy on Openshift , with standalone Spark cluster

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.6.0
• Fix Version/s: 0.9.0
• Component/s: Batch, Server
• Labels: LIVY livy livy.conf openshift spark standalone
• Environment: Openshift 3.11 on-premise, Spark 2.3.1 - cluster mode (standalone)
• Assignee: Unassigned
• Reporter: Matar Peles
• Created: 18/Sep/19 15:16
• Updated: 12/Nov/22 22:10

Description

Trying to submit a Spark batch job using Apache Livy. I am running my Spark cluster and Livy service on Openshift. the Spark cluster is in standalone cluster mode, Livy runs spark in cluster deploy mode. I've configured Livy to the Spark cluster master service:

livy.spark.master = "spark://spark.spark.svc:7077

when i try to run a scala spark job, using the POST /batches method and get the exception:

Unable to connect to server spark://spark.spark.svc:7077