I'm proposing to do the following:
1. Short term fix for 2.7.1: Check if service address in timeline DT is empty or not. If empty, we fall back to use the configured service address. It will make app submission via REST API work in secure mode without additional DT process work unless users really want to renew the DT from somewhere other than the configure address. It shouldn't be common as we usually only setup one timeline server per YARN cluster.
2. Long term fix: we can do something similar to
HDFS-6904. Let the client to pass in the service address, and set token's service address at server side before serializing it into a string. And this problem is not just limited to ATS. RM REST API doesn't set the service address for RM DT too. It's better to seek for a common solution. For example, we can fix DelegationTokenAuthenticationHandler to make all use cases of hadoop http auth component set the service addr properly. One step further, even RPC protocol may have the similar problem. For example, if we work with ApplicationClientProtocol directly, we should get an RM DT without service address (correct me if I'm wrong).