Major Description
As reported by atm :
The issue is due to the authentication token that the NN has to talk to the KMS is expiring, AND the signature secret provider in the KMS authentication filter is discarding the old secret after 2x the authentication token validity period.
If the token being supplied is under 1x the validity lifetime then the token will authenticate just fine. If the token being supplied is between 1x-2x the validity lifetime, then the token can be validated but it will be expired, so a 401 will be returned to the client and it will get a new token. But if the token being supplied is greater than 2x the validity lifetime, then the KMS authentication filter will not even be able to validate the token, and will return a 403, which will cause the client to not retry authentication to the KMS.
The KMSClientProvider needs to be modified to retry authentication even in the above case