Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Won't Fix
-
1.0.0
-
None
-
None
-
None
Description
in spark-class and spark-submit
the SPARK_HOME is set to the present working directory, causing the value of already defined SPARK_HOME being overwritten
we should not overwrite that if SPARK_HOME has been defined
Our scenario
we have a login portal for all the team members to use the spark-cluster, everyone gets an account and home directory
spark-1.0 is copied in the root path "/", every account gets a soft link to the /spark-1.0 in it's home directory
spark 1.0 is deployed in a cluster, where the user name is different with any accounts except one, say "nanzhu", in login portal,
so when the user runs spark-shell, it always tries to run /home/user_account/spark-1.0/bin/compute-class.sh, which does not exist.
We set a global SPARK_HOME to /home/nanzhu/spark-1.0 globally which is consistent with the remote cluster setup, but unfortunately, this is overwritten by the spark-class and spark-submit ....
