Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-11413

SolrGraphiteReporter fails to report metrics due to non-thread safe code

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 6.6, 7.0
    • Fix Version/s: 7.2, 8.0
    • Component/s: metrics
    • Labels:
      None

      Description

      Symptom:
      Intermittent errors writing graphite metrics. Errors indicate use of sockets which have already been closed.

      Cause:
      SolrGraphiteReporter caches and shares dropwizard Graphite instances. These reporters are not thread safe as they open and close an instance variable of type GraphiteSender. On modern bare metal hardware this problem was observed consistently, and resulted in the majority of metrics failing to be delivered to graphite.

      Proposed Fix:
      Graphite (and PickledGraphite) are not designed to be cached, and should not be.

      Test:
      Patch file includes test which forces error.

      Alternative Fixes Considered:

      • Totally change solr metrics architecture to use a single metrics registry - seems undesirable and impractical
      • Create a synchronized or otherwise thread-safe implementation of dropwizard graphite reporter - should be fixed upstream in dropwizard and not obviously preferred to current model

        Attachments

        1. SOLR-11413.patch
          1 kB
          Erik Persson

          Activity

            People

            • Assignee:
              ab Andrzej Bialecki
              Reporter:
              erikpersson Erik Persson
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: