[BIGTOP-3462] Deploying Livy fails if the SPARK_HOME directory doesn't exist

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: deployment
• Assignee: Kengo Seki
• Reporter: Kengo Seki
• Created: 04/Dec/20 13:32
• Updated: 06/Dec/20 05:42
• Resolved: 06/Dec/20 05:42

Description

On some distros/platforms, deploying Livy fails with the following messages.

Error: Systemd start for livy-server failed!
journalctl log for livy-server:
-- Logs begin at Thu 2020-12-03 09:48:16 UTC, end at Thu 2020-12-03 10:01:07 UTC. --
Dec 03 10:00:59 59fa1cab214f systemd[1]: Starting LSB: Livy Server...
Dec 03 10:00:59 59fa1cab214f runuser[4287]: pam_unix(runuser:session): session opened for user livy by (uid=0)
Dec 03 10:00:59 59fa1cab214f livy-server[4270]: starting /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.272.b10-1.el8_2.aarch64/bin/java  -cp /usr/lib/livy/jars/*:/usr/lib/livy/conf:/etc/hadoop/conf: org.apache.livy.server.LivyServer, logging to /var/log/livy/livy-livy-server.out
Dec 03 10:01:01 59fa1cab214f runuser[4287]: pam_unix(runuser:session): session closed for user livy
Dec 03 10:01:06 59fa1cab214f livy-server[4270]: Failed to start Livy Server. Return value: 1[FAILED]
Dec 03 10:01:06 59fa1cab214f systemd[1]: livy-server.service: Control process exited, code=exited status=1
Dec 03 10:01:06 59fa1cab214f systemd[1]: livy-server.service: Failed with result 'exit-code'.
Dec 03 10:01:06 59fa1cab214f systemd[1]: Failed to start LSB: Livy Server.

https://ci.bigtop.apache.org/job/Bigtop-1.5.0-smoke-tests/COMPONENTS=hdfs.yarn.spark.livy@livy,OS=centos-8-aarch64-deploy/2/console
https://ci.bigtop.apache.org/job/Bigtop-1.5.0-smoke-tests/COMPONENTS=hdfs.yarn.spark.livy@livy,OS=centos-8-x86_64-deploy/2/console
https://ci.bigtop.apache.org/job/Bigtop-1.5.0-smoke-tests/COMPONENTS=hdfs.yarn.spark.livy@livy,OS=fedora-31-aarch64-deploy/2/console
https://ci.bigtop.apache.org/job/Bigtop-1.5.0-smoke-tests/COMPONENTS=hdfs.yarn.spark.livy@livy,OS=fedora-31-x86_64-deploy/2/console

So I reproduced it locally and found the following error in /var/log/livy/livy-livy-server.out, referred to above:

$ docker exec -it 20201204_103251_r9202_bigtop_1 cat /var/log/livy/livy-livy-server.out
20/12/04 10:40:51 INFO AccessManager: AccessControlManager acls disabled;users with view permission: ;users with modify permission: ;users with super permission: ;other allowed users: *
Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: SPARK_HOME path does not exist
	at scala.Predef$.require(Predef.scala:224)
	at org.apache.livy.utils.LivySparkUtils$.testSparkHome(LivySparkUtils.scala:56)
	at org.apache.livy.server.LivyServer.start(LivyServer.scala:77)
	at org.apache.livy.server.LivyServer$.main(LivyServer.scala:423)
	at org.apache.livy.server.LivyServer.main(LivyServer.scala)

To avoid this situation, Livy server should be started after the spark-core package is installed.

Issue Links

• links to: https://github.com/apache/bigtop/pull/709