Ok, apparently spark.master yarn will only affect submitted applications and it needs to be fixed for non-yarn deployments like standalone, etc. Perhaps it might be done as a separate JIRA or as an improvement for the Puppet recipes update.
In the meanwhile, I have figured out what was wrong with the startup (although I don't know why). For some reason, master is binding to the container's IP address (e.g. 172.17.0.10) despite that SPARK_MASTER_IP is set to the container's hostname. Work however is trying to connect the master via the hostname and the whole shenanigan fails ;(
It doesn't seem like a packaging issue, but more like a screwup on the Spark side. Yet, we still need to figure out why this is happening. I anyone has ideas - please chime in!
I tend to commit this patch tomorrow afternoon unless I hear otherwise from the author or someone else.