it won't have any benefit to give client flexibility to be different with what server settings. Isn't it?
Agreed, I think the best approach here long-term is to not have this be a config setting at all but rather derived from the communication session that retrieved the token in the first place. I also agree that it is going to be more dangerous than not for job.xml to override this particular setting (whether or not we're using an MR tarball).
if we think everything inside of MR tarball should be per job only
This is definitely not the case, at least for our clusters. As I mentioned above, the MR tarballs are created by the cluster admins and therefore have the appropriate configs for that cluster. We do not want the jobs to pick up configs from the nodes per the issues I described earlier.
I am not fully agree that it should be cluster admin' job to create tarball and keep consistent for all configurations with cluster settings.
The tarball consist of the Hadoop jars and the *-site.xml files. Both of these are things admins of the cluster are expected to maintain and provide. Therefore I think it's completely appropriate that typically these tarballs are created and provided by admins. We're not running a bunch of different versions of the tarball for different types of jobs. We just have the main tarball for the cluster and users override various settings for their specific job via job.xml, just like a regular non-tarball job does.
To be clear, I'm not saying everyone has to run it the way we do. However if you don't then there needs to be solutions for the types of rolling upgrade problems I pointed out above. Also I don't think the way we're using the tarball is an invalid setup, so therefore any proposed change needs to ensure this type of setup keeps working.