Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-13234

Prometheus Metric Exporter Not Threadsafe

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      The Solr Prometheus Exporter collects metrics when it receives a HTTP request from Prometheus. Prometheus sends this request, on its scrape interval. When the time taken to collect the Solr metrics is greater than the scrape interval of the Prometheus server, this results in concurrent metric collection occurring in this method. This method doesn’t appear to be thread safe, for instance you could have concurrent modifications of a map. After a while the Solr Exporter processes becomes nondeterministic, we've observed NPE and loss of metrics.

      To address this, I'm proposing the following fixes:

      1. Read/parse the configuration at startup and make it immutable. 
      2. Collect metrics from Solr on an interval which is controlled by the Solr Exporter and cache the metric samples to return during Prometheus scraping. Metric collection can be expensive, for example executing arbitrary Solr searches, it's not ideal to allow for concurrent metric collection and on an interval which is not defined by the Solr Exporter.

      There are also a few other performance improvements that we've made while fixing this, for example using the ClusterStateProvider instead of sending multiple HTTP requests to each Solr node to lookup all the cores.

      I'm currently finishing up these changes which I'll submit as a PR.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            shalin Shalin Shekhar Mangar
            danyal Danyal Prout
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 40m
                40m

                Slack

                  Issue deployment