Major Description
When new security plugins are initialized due to remote API requetss, there is a delay between "registering" the new plugins for use in methods like initializeAuthenticationPlugin() (by assigning them to CoreContainer's volatile this.authenticationPlugin variable) and when the initializeMetrics(..) method is called on these plugins, so that they continue to use the existing Metric instances as the plugins they are replacing.
Because these security plugins maintain local refrences to these Metrics (and don't "get" them from the MetricRegisty everytime they need to inc() them) this means that there is short race condition situation such that during the window of time after a new plugin instance is put into use, but before initializeMetrics(..) is called on them, at these plugins are responsible for accepting/rejecting requests, and decisions in Metric instances that are not registered and subsequently get thrown away (and GCed) once the CoreContainer gets around to calling initializeMetrics(..) (and the plugin starts using the pre-existing metric objects)
This has some noticible impacts on auth tests on CPU constrained jenkins machines (even after putting in place SOLR-13464 work arounds) that make assertions about the metrics recorded.
In real world situations, the impact of this bug on users is minor: for a few micro/milli-seconds, requests may come in w/o being counted in the auth metrics – which may also result in descrepencies between the auth metrics totals and the overall request metrics.