Hi Anu Engineer - I appreciate your concerns and believe that the third option that I mentioned may provide a solution for the inconsistency. At the same time provide sane defaults that will allow all clients that are not vulnerable to the attack and should not be subject to the requirements imposed on browsers to continue to work.
If we reverse the list of non-browsers into a list of browser matching regex expressions then all non-browsers will continue to work.
The challenge will be to identify the proper set of regex expressions to protect the vast majority of browsers.
I don't see the requirement that browsers/AJAX provide additional HTTP headers as a change to the REST protocol. This is an access requirement between the browser client and the server and does not involve the semantics of the REST API itself. It is really an application level concern. Similar to authentication mechanisms differing between a web app and a curl invocation.
So, let's walk through the option 3 description:
1. default set of browser matching regex expressions in place
2. all non-browsers continue to work without requiring a header to protect them from an attack that they are not vulnerable to
3. AJAX calls are changed to send the header based on configured header names or default
4. user-agents that match a regex pattern have their header validated as being present and therefore from a valid AJAX client
5. user-agents that do not match a regex pattern do not have their header validated and therefore are vulnerable to CSRF
#5 above is the concerning bit that is introduced by this option 3.
We might be able to discern some additional client context from the Referrer header which also cannot be altered by AJAX calls.
This will require some investigation into what Referrer is (if anything at all) for non-browsers.
The idea being something like...
1. If #5 and there is no referrer then we don't validate the header or
2. if #5 and the referrer matches some additional config for allowed referrers then validate the existence of a the header (this would indicate the use of an unusual browser and should be added to regex patterns)
That level could be a follow up JIRA since given a sane set of browser regex patterns as defaults this should be an edge case.