What happens when both of these are enabled at the same time with different settings?
If the filter is set in core-site.xml, it will be enabled irrespective of the value set in yarn.timeline-service.http-cross-origin.enabled. There is no change in behavior here - if a user sets the existing timeline CORS filter in core-site.xml, it will override the value set in yarn.timeline-service.http-cross-origin.enabled.
Is there a need to allow the design to enable this only for webservices (REST APIs) instead of the whole webserver (builtin UIs and REST apis)?
I don't think there's any such need. As far as I can tell, we don't treat http requests differently i.e. webservices and builtin UI requests are treated the same.
Not sure if there is a question of selecting enabling cors support for different services such as NN webservices vs RM webservices.
I thought about this - I'm not sure there's any benefit to adding one more set of config knobs. The chances are if you're fine with enabling CORS for the RM, you're probably fine enabling it for the NMs and the timeline server as well. If a user wishes to disable or enable CORS for a particular service, they can always set the filter initializers to the appropriate value on the node the service is running on(either via core-site.xml or yarn-site.xml).