There are two related problems with spark.mesos.secret:
1) The web UI shows its value in the "environment" tab
2) Passing it as a command-line option to spark-submit (or creating a SparkContext from python, with the effect of launching spark-submit) exposes it to "ps"
I'll be happy to submit a patch but I could use some advice first.
The first problem is easy enough, just don't show that value in the UI
For the second problem, I'm not sure what the best solution is. A "spark.mesos.secret-file" parameter would let the user store the secret in a non-world-readable file. Alternatively, the mesos secret could be obtained from the environment, which other users don't have access to. Either solution would work in client mode, but I don't know if they're workable in cluster mode.