Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9854

Collect metrics for index merges and index store IO

        Details

        • Type: Improvement
        • Status: Resolved
        • Priority: Minor
        • Resolution: Fixed
        • Affects Version/s: 6.4, 7.0
        • Fix Version/s: 6.4, 7.0
        • Component/s: metrics
        • Security Level: Public (Default Security Level. Issues are Public)
        • Labels:
          None

          Description

          Using API for metrics management developed in SOLR-4735 we should also start collecting metrics for major aspects of IndexWriter operation, such as read / write IO rates, number of minor and major merges and IO during these operations, etc.

          This will provide a better insight into resource consumption and load at the IO level.

          1. Text File
            SOLR-9854.patch
            54 kB
            Andrzej Bialecki
          2. Text File
            SOLR-9854.patch
            37 kB
            Andrzej Bialecki

            Issue Links

            breaks

            Bug - A problem which impairs or prevents the functions of the product. SOLR-10130 Serious performance degradation in Solr 6.4.1 due to the new metrics collection

            • Blocker - Blocks development and/or testing work, production could not run
            • Resolved
            is duplicated by

            Improvement - An improvement or enhancement to an existing feature or task. SOLR-1550 statistics for request handlers should report std dev

            • Trivial - Cosmetic problem like misspelt words or misaligned text.
            • Resolved
            is required by

            Improvement - An improvement or enhancement to an existing feature or task. SOLR-9857 Collect aggregated metrics from replicas in shard leader

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Resolved
            requires

            Improvement - An improvement or enhancement to an existing feature or task. SOLR-4735 Improve Solr metrics reporting

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Resolved

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              ab Andrzej Bialecki added a comment - - edited

              Patch relative to master. See the details on branch jira/solr-9854, here's a short log of key commits from this branch:

              commit 6ceb05074b3f6303e660613e277c3e26472ec627
Author: Andrzej Bialecki <ab@apache.org>
Date:   Fri Dec 23 21:30:15 2016 +0100

    SOLR-9854 Move metrics collection to SolrIndexWriter to get more meaningful
    timings of merges. Other changes:
    * simplify metric names in MetricUtils. Don't scale units for non-timer histograms
      as they may represent other units (eg. bytes)
    * add ability to configure metric details and parameters (currently the threshold
      between minor and major merges).
...
commit ce8ab8c6359753623ae7a96204badd16193efc09
Author: Andrzej Bialecki <ab@apache.org>
Date:   Thu Dec 15 13:33:28 2016 +0100

    SOLR-9854 Initial version. This adds metrics to Directory IO and also
    monitors minor / major merge operations.
              Show
              ab Andrzej Bialecki added a comment - - edited Patch relative to master. See the details on branch jira/solr-9854 , here's a short log of key commits from this branch: commit 6ceb05074b3f6303e660613e277c3e26472ec627 Author: Andrzej Bialecki <ab@apache.org> Date: Fri Dec 23 21:30:15 2016 +0100 SOLR-9854 Move metrics collection to SolrIndexWriter to get more meaningful timings of merges. Other changes: * simplify metric names in MetricUtils. Don't scale units for non-timer histograms as they may represent other units (eg. bytes) * add ability to configure metric details and parameters (currently the threshold between minor and major merges). ... commit ce8ab8c6359753623ae7a96204badd16193efc09 Author: Andrzej Bialecki <ab@apache.org> Date: Thu Dec 15 13:33:28 2016 +0100 SOLR-9854 Initial version. This adds metrics to Directory IO and also monitors minor / major merge operations.
              Hide
              Permalink
              ab Andrzej Bialecki added a comment -

              Current patch. I think this is ready.

              Show
              ab Andrzej Bialecki added a comment - Current patch. I think this is ready.
              Hide
              Permalink
              jira-bot ASF subversion and git services added a comment -

              Commit 48ca9fc3f4f8d95293cee7bb59eff61247ede181 in lucene-solr's branch refs/heads/master from Andrzej Bialecki
              [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=48ca9fc ]

              SOLR-9854: Collect metrics for index merges and index store IO. (squashed)

              Show
              jira-bot ASF subversion and git services added a comment - Commit 48ca9fc3f4f8d95293cee7bb59eff61247ede181 in lucene-solr's branch refs/heads/master from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=48ca9fc ] SOLR-9854 : Collect metrics for index merges and index store IO. (squashed)
              Hide
              Permalink
              jira-bot ASF subversion and git services added a comment -

              Commit c40cd2df49c80aee1ab2b6fea634191edc8b944f in lucene-solr's branch refs/heads/branch_6x from Andrzej Bialecki
              [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c40cd2d ]

              SOLR-9854: Collect metrics for index merges and index store IO (squashed).

              Show
              jira-bot ASF subversion and git services added a comment - Commit c40cd2df49c80aee1ab2b6fea634191edc8b944f in lucene-solr's branch refs/heads/branch_6x from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c40cd2d ] SOLR-9854 : Collect metrics for index merges and index store IO (squashed).
              Hide
              Permalink
              jira-bot ASF subversion and git services added a comment -

              Commit b9827bcba9ee4e1b0dea8a18c6efe4322a400b09 in lucene-solr's branch refs/heads/master from Andrzej Bialecki
              [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b9827bc ]

              SOLR-9854 Relax test assertions.

              Show
              jira-bot ASF subversion and git services added a comment - Commit b9827bcba9ee4e1b0dea8a18c6efe4322a400b09 in lucene-solr's branch refs/heads/master from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b9827bc ] SOLR-9854 Relax test assertions.
              Hide
              Permalink
              jira-bot ASF subversion and git services added a comment -

              Commit 03a41258b13075ff2fd305f538c82241503d7821 in lucene-solr's branch refs/heads/branch_6x from Andrzej Bialecki
              [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=03a4125 ]

              SOLR-9854 Relax test assertions.

              Show
              jira-bot ASF subversion and git services added a comment - Commit 03a41258b13075ff2fd305f538c82241503d7821 in lucene-solr's branch refs/heads/branch_6x from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=03a4125 ] SOLR-9854 Relax test assertions.

                People

                • Assignee:
                  ab Andrzej Bialecki
                  Reporter:
                  ab Andrzej Bialecki
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development