Major Description
Symptom:
Intermittent errors writing graphite metrics. Errors indicate use of sockets which have already been closed.
Cause:
SolrGraphiteReporter caches and shares dropwizard Graphite instances. These reporters are not thread safe as they open and close an instance variable of type GraphiteSender. On modern bare metal hardware this problem was observed consistently, and resulted in the majority of metrics failing to be delivered to graphite.
Proposed Fix:
Graphite (and PickledGraphite) are not designed to be cached, and should not be.
Test:
Patch file includes test which forces error.
Alternative Fixes Considered:
- Totally change solr metrics architecture to use a single metrics registry - seems undesirable and impractical
- Create a synchronized or otherwise thread-safe implementation of dropwizard graphite reporter - should be fixed upstream in dropwizard and not obviously preferred to current model