Details
-
Bug
-
Status: Open
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
It looks like the TOREE-97 issue – support for Spark Yarn was closed without definitive solution (or something went wrong on the way). Toree does support it, but it won't work if a user doesn't add manually in their kernel.json definition, the env vars for HADOOP_CONF_DIR. Without that env var, Spark doesn't know what to do with the option --master=yarn (set in _TOREE_SPARK_OPTS_). It would be desirable to have it by default.
Probably this is not the nicest way to solve the problem, because it just hard codes more vars into the JSON file – ideally it would be nice to have an interface to add or remove env vars from those files, however, HADOOP_CONF_DIR and SPARK_CONF_DIR look basic to be exported. Even for an Spark Standalone deployment, HADOOP_CONF_DIR won't hurt. So, here it goes our 2 cents to improve a bit the situation.