keep all auth config in one place
Sure, but it's called "security-site.xml", but it doesn't contain most of the Hadoop security configs, just the HTTP auth filter configs. For that matter, is it unreasonable that there might be some HTTP auth filter config which clients might need to know? It seems like we're unnecessarily overloading the purpose of this file to be both secret and to group like-configs in the same place.
make sense. Do we have such group today?
The answer to that is going to be specific to whatever method a user uses to install/configure Hadoop. FWIW, in CDH's packages there is a 'hadoop' group which both 'mapred' and 'hdfs' belong to. I don't know for sure, but I bet this is what the built-in Hadoop packages do, too.
so where would be the location of this 'secret' file?
Good question. Kerberos has two distinct files: /etc/krb5.conf and /etc/krb5kdc/kdc.conf, where the former is world-readable, and the latter is not. Maybe, then, /etc/hadoop/conf/ and /etc/hadoop/conf-secret/ ?
AFAIK you can set permissions with File but you cannot check them.
Ah, that could very well be. Pretty sure Hadoop has some classes with helper methods to deal with getting/setting group permissions, which might use JNI or fork a sub-process.
using a 'secret' file just for this secret, what if there is something else needing a secret? we don't want those 'secrets' files to proliferate.
That's totally valid, and a very good point. Still, though, I'm not sure that the solution you have here is appropriate. For example, what if something should be secret only to the MR daemons, but not to the HDFS daemons, or vice versa? We couldn't reasonably put such a secret in security-site.xml, since as previously-mentioned this file must be readable by both.