Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9805

Use metrics-jvm library to instrument jvm internals

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.4, 7.0
    • Component/s: metrics
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None
    1. jvm-metrics.xml
      7 kB
      Andrzej Bialecki
    2. SOLR-9805.patch
      5 kB
      Shalin Shekhar Mangar

      Issue Links

        Activity

        Hide
        shalinmangar Shalin Shekhar Mangar added a comment -

        Patch which applies to the features/metrics branch. We add all jvm metric sets from metrics-jvm library to a shared metric registry named "/jvm". Following are the metrics collected:

        {
          "version" : "3.0.0",
          "gauges" : {
            "ConcurrentMarkSweep.count" : {
              "value" : 0
            },
            "ConcurrentMarkSweep.time" : {
              "value" : 6
            },
            "ParNew.count" : {
              "value" : 1
            },
            "ParNew.time" : {
              "value" : 10
            },
            "blocked.count" : {
              "value" : 0
            },
            "count" : {
              "value" : 21
            },
            "daemon.count" : {
              "value" : 7
            },
            "deadlock.count" : {
              "value" : 0
            },
            "deadlocks" : {
              "value" : [ ]
            },
            "direct.capacity" : {
              "value" : 57344
            },
            "direct.count" : {
              "value" : 4
            },
            "direct.used" : {
              "value" : 57344
            },
            "fileDescriptorRatio" : {
              "value" : 0.0037841796875
            },
            "heap.committed" : {
              "value" : 514523136
            },
            "heap.init" : {
              "value" : 536870912
            },
            "heap.max" : {
              "value" : 514523136
            },
            "heap.usage" : {
              "value" : 0.07437098416503471
            },
            "heap.used" : {
              "value" : 38265592
            },
            "loaded" : {
              "value" : 4179
            },
            "mapped.capacity" : {
              "value" : 0
            },
            "mapped.count" : {
              "value" : 0
            },
            "mapped.used" : {
              "value" : 0
            },
            "new.count" : {
              "value" : 0
            },
            "non-heap.committed" : {
              "value" : 34381824
            },
            "non-heap.init" : {
              "value" : 2555904
            },
            "non-heap.max" : {
              "value" : -1
            },
            "non-heap.usage" : {
              "value" : -3.3637024E7
            },
            "non-heap.used" : {
              "value" : 33637024
            },
            "pools.CMS-Old-Gen.committed" : {
              "value" : 402653184
            },
            "pools.CMS-Old-Gen.init" : {
              "value" : 402653184
            },
            "pools.CMS-Old-Gen.max" : {
              "value" : 402653184
            },
            "pools.CMS-Old-Gen.usage" : {
              "value" : 0.0
            },
            "pools.CMS-Old-Gen.used" : {
              "value" : 0
            },
            "pools.Code-Cache.committed" : {
              "value" : 6160384
            },
            "pools.Code-Cache.init" : {
              "value" : 2555904
            },
            "pools.Code-Cache.max" : {
              "value" : 251658240
            },
            "pools.Code-Cache.usage" : {
              "value" : 0.024188232421875
            },
            "pools.Code-Cache.used" : {
              "value" : 6087872
            },
            "pools.Compressed-Class-Space.committed" : {
              "value" : 3092480
            },
            "pools.Compressed-Class-Space.init" : {
              "value" : 0
            },
            "pools.Compressed-Class-Space.max" : {
              "value" : 1073741824
            },
            "pools.Compressed-Class-Space.usage" : {
              "value" : 0.0027397125959396362
            },
            "pools.Compressed-Class-Space.used" : {
              "value" : 2941744
            },
            "pools.Metaspace.committed" : {
              "value" : 25128960
            },
            "pools.Metaspace.init" : {
              "value" : 0
            },
            "pools.Metaspace.max" : {
              "value" : -1
            },
            "pools.Metaspace.usage" : {
              "value" : 0.9796426115525673
            },
            "pools.Metaspace.used" : {
              "value" : 24617400
            },
            "pools.Par-Eden-Space.committed" : {
              "value" : 89522176
            },
            "pools.Par-Eden-Space.init" : {
              "value" : 89522176
            },
            "pools.Par-Eden-Space.max" : {
              "value" : 89522176
            },
            "pools.Par-Eden-Space.usage" : {
              "value" : 0.2822161963534041
            },
            "pools.Par-Eden-Space.used" : {
              "value" : 25264608
            },
            "pools.Par-Survivor-Space.committed" : {
              "value" : 22347776
            },
            "pools.Par-Survivor-Space.init" : {
              "value" : 22347776
            },
            "pools.Par-Survivor-Space.max" : {
              "value" : 22347776
            },
            "pools.Par-Survivor-Space.usage" : {
              "value" : 0.581757397246151
            },
            "pools.Par-Survivor-Space.used" : {
              "value" : 13000984
            },
            "runnable.count" : {
              "value" : 9
            },
            "terminated.count" : {
              "value" : 0
            },
            "timed_waiting.count" : {
              "value" : 9
            },
            "total.committed" : {
              "value" : 548904960
            },
            "total.init" : {
              "value" : 539426816
            },
            "total.max" : {
              "value" : 514523135
            },
            "total.used" : {
              "value" : 71930344
            },
            "unloaded" : {
              "value" : 0
            },
            "waiting.count" : {
              "value" : 3
            }
          },
          "counters" : { },
          "histograms" : { },
          "meters" : { },
          "timers" : { }
        }
        

        Since the /admin/metrics endpoint is constrained to show only one metric registry at a time, it does not show the JVM metrics. I'll fix the metrics API in a separate issue.

        Show
        shalinmangar Shalin Shekhar Mangar added a comment - Patch which applies to the features/metrics branch. We add all jvm metric sets from metrics-jvm library to a shared metric registry named "/jvm". Following are the metrics collected: { "version" : "3.0.0" , "gauges" : { "ConcurrentMarkSweep.count" : { "value" : 0 }, "ConcurrentMarkSweep.time" : { "value" : 6 }, "ParNew.count" : { "value" : 1 }, "ParNew.time" : { "value" : 10 }, "blocked.count" : { "value" : 0 }, "count" : { "value" : 21 }, "daemon.count" : { "value" : 7 }, "deadlock.count" : { "value" : 0 }, "deadlocks" : { "value" : [ ] }, "direct.capacity" : { "value" : 57344 }, "direct.count" : { "value" : 4 }, "direct.used" : { "value" : 57344 }, "fileDescriptorRatio" : { "value" : 0.0037841796875 }, "heap.committed" : { "value" : 514523136 }, "heap.init" : { "value" : 536870912 }, "heap.max" : { "value" : 514523136 }, "heap.usage" : { "value" : 0.07437098416503471 }, "heap.used" : { "value" : 38265592 }, "loaded" : { "value" : 4179 }, "mapped.capacity" : { "value" : 0 }, "mapped.count" : { "value" : 0 }, "mapped.used" : { "value" : 0 }, " new .count" : { "value" : 0 }, "non-heap.committed" : { "value" : 34381824 }, "non-heap.init" : { "value" : 2555904 }, "non-heap.max" : { "value" : -1 }, "non-heap.usage" : { "value" : -3.3637024E7 }, "non-heap.used" : { "value" : 33637024 }, "pools.CMS-Old-Gen.committed" : { "value" : 402653184 }, "pools.CMS-Old-Gen.init" : { "value" : 402653184 }, "pools.CMS-Old-Gen.max" : { "value" : 402653184 }, "pools.CMS-Old-Gen.usage" : { "value" : 0.0 }, "pools.CMS-Old-Gen.used" : { "value" : 0 }, "pools.Code-Cache.committed" : { "value" : 6160384 }, "pools.Code-Cache.init" : { "value" : 2555904 }, "pools.Code-Cache.max" : { "value" : 251658240 }, "pools.Code-Cache.usage" : { "value" : 0.024188232421875 }, "pools.Code-Cache.used" : { "value" : 6087872 }, "pools.Compressed- Class -Space.committed" : { "value" : 3092480 }, "pools.Compressed- Class -Space.init" : { "value" : 0 }, "pools.Compressed- Class -Space.max" : { "value" : 1073741824 }, "pools.Compressed- Class -Space.usage" : { "value" : 0.0027397125959396362 }, "pools.Compressed- Class -Space.used" : { "value" : 2941744 }, "pools.Metaspace.committed" : { "value" : 25128960 }, "pools.Metaspace.init" : { "value" : 0 }, "pools.Metaspace.max" : { "value" : -1 }, "pools.Metaspace.usage" : { "value" : 0.9796426115525673 }, "pools.Metaspace.used" : { "value" : 24617400 }, "pools.Par-Eden-Space.committed" : { "value" : 89522176 }, "pools.Par-Eden-Space.init" : { "value" : 89522176 }, "pools.Par-Eden-Space.max" : { "value" : 89522176 }, "pools.Par-Eden-Space.usage" : { "value" : 0.2822161963534041 }, "pools.Par-Eden-Space.used" : { "value" : 25264608 }, "pools.Par-Survivor-Space.committed" : { "value" : 22347776 }, "pools.Par-Survivor-Space.init" : { "value" : 22347776 }, "pools.Par-Survivor-Space.max" : { "value" : 22347776 }, "pools.Par-Survivor-Space.usage" : { "value" : 0.581757397246151 }, "pools.Par-Survivor-Space.used" : { "value" : 13000984 }, "runnable.count" : { "value" : 9 }, "terminated.count" : { "value" : 0 }, "timed_waiting.count" : { "value" : 9 }, "total.committed" : { "value" : 548904960 }, "total.init" : { "value" : 539426816 }, "total.max" : { "value" : 514523135 }, "total.used" : { "value" : 71930344 }, "unloaded" : { "value" : 0 }, "waiting.count" : { "value" : 3 } }, "counters" : { }, "histograms" : { }, "meters" : { }, "timers" : { } } Since the /admin/metrics endpoint is constrained to show only one metric registry at a time, it does not show the JVM metrics. I'll fix the metrics API in a separate issue.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 8601477a65ba9141a39389341230b273a69fcecc in lucene-solr's branch refs/heads/feature/metrics from Shalin Shekhar Mangar
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8601477 ]

        SOLR-9805: Use metrics-jvm library to instrument jvm internals such as GC, memory usage and others

        Show
        jira-bot ASF subversion and git services added a comment - Commit 8601477a65ba9141a39389341230b273a69fcecc in lucene-solr's branch refs/heads/feature/metrics from Shalin Shekhar Mangar [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8601477 ] SOLR-9805 : Use metrics-jvm library to instrument jvm internals such as GC, memory usage and others
        Hide
        shalinmangar Shalin Shekhar Mangar added a comment -

        The metric registry name has been changed to "solr.jvm" in the above commit.

        Show
        shalinmangar Shalin Shekhar Mangar added a comment - The metric registry name has been changed to "solr.jvm" in the above commit.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 54e35102fe0d18f8a14b3cbd1d368c5d47cfb706 in lucene-solr's branch refs/heads/feature/metrics from Shalin Shekhar Mangar
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=54e3510 ]

        SOLR-9805: Added sha, notice and license files for metrics-jvm library

        Show
        jira-bot ASF subversion and git services added a comment - Commit 54e35102fe0d18f8a14b3cbd1d368c5d47cfb706 in lucene-solr's branch refs/heads/feature/metrics from Shalin Shekhar Mangar [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=54e3510 ] SOLR-9805 : Added sha, notice and license files for metrics-jvm library
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 8bbdb6248c5de3f3bd61501ba42a50aeec29c78b in lucene-solr's branch refs/heads/master from Andrzej Bialecki
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8bbdb62 ]

        Squashed commit of branch 'feature/metrics', containing:
        SOLR-4735: Improve Solr metrics reporting
        SOLR-9812: Implement /admin/metrics API
        SOLR-9805: Use metrics-jvm library to instrument jvm internals
        SOLR-9788: Use instrumented jetty classes

        Show
        jira-bot ASF subversion and git services added a comment - Commit 8bbdb6248c5de3f3bd61501ba42a50aeec29c78b in lucene-solr's branch refs/heads/master from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8bbdb62 ] Squashed commit of branch 'feature/metrics', containing: SOLR-4735 : Improve Solr metrics reporting SOLR-9812 : Implement /admin/metrics API SOLR-9805 : Use metrics-jvm library to instrument jvm internals SOLR-9788 : Use instrumented jetty classes
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 80462df86efaf093fdebeac5eef1727c5fda968f in lucene-solr's branch refs/heads/master from Andrzej Bialecki
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=80462df ]

        SOLR-9805 Don't use FileDescriptorRatioGauge - internally it uses reflection and
        doesn't work under Java 9. Instead use this opportunity to implement a more
        detailed OperatingSystemMetricSet. Add a unit test. Simplify some of the metric names.

        Show
        jira-bot ASF subversion and git services added a comment - Commit 80462df86efaf093fdebeac5eef1727c5fda968f in lucene-solr's branch refs/heads/master from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=80462df ] SOLR-9805 Don't use FileDescriptorRatioGauge - internally it uses reflection and doesn't work under Java 9. Instead use this opportunity to implement a more detailed OperatingSystemMetricSet. Add a unit test. Simplify some of the metric names.
        Hide
        ab Andrzej Bialecki added a comment -

        Example results from /admin/metrics?group=jvm.

        Show
        ab Andrzej Bialecki added a comment - Example results from /admin/metrics?group=jvm .
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 687f03661d726d70d5718db8b45990b3fbc98183 in lucene-solr's branch refs/heads/master from Andrzej Bialecki
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=687f036 ]

        SOLR-9805 Fix NPE.

        Show
        jira-bot ASF subversion and git services added a comment - Commit 687f03661d726d70d5718db8b45990b3fbc98183 in lucene-solr's branch refs/heads/master from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=687f036 ] SOLR-9805 Fix NPE.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 56476fb8c70486c21324879f1bd1efc79df92ffb in lucene-solr's branch refs/heads/master from Andrzej Bialecki
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=56476fb ]

        SOLR-9805 Fix assertion error on Windows where SystemLoadAverage is reported as -1.
        Don't expose non-existent attributes.

        Show
        jira-bot ASF subversion and git services added a comment - Commit 56476fb8c70486c21324879f1bd1efc79df92ffb in lucene-solr's branch refs/heads/master from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=56476fb ] SOLR-9805 Fix assertion error on Windows where SystemLoadAverage is reported as -1. Don't expose non-existent attributes.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 9dde8a30303de4bce5da45189219dd6150252b29 in lucene-solr's branch refs/heads/branch_6x from Andrzej Bialecki
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9dde8a3 ]

        Cumulative patch from master, originally developed on branch
        'feature/metrics', which brings the following issues:

        • SOLR-4735: Improve Solr metrics reporting
        • SOLR-9812: Implement /admin/metrics API
        • SOLR-9805: Use metrics-jvm library to instrument jvm internals
        • SOLR-9788: Use instrumented jetty classes
        Show
        jira-bot ASF subversion and git services added a comment - Commit 9dde8a30303de4bce5da45189219dd6150252b29 in lucene-solr's branch refs/heads/branch_6x from Andrzej Bialecki [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9dde8a3 ] Cumulative patch from master, originally developed on branch 'feature/metrics', which brings the following issues: SOLR-4735 : Improve Solr metrics reporting SOLR-9812 : Implement /admin/metrics API SOLR-9805 : Use metrics-jvm library to instrument jvm internals SOLR-9788 : Use instrumented jetty classes

          People

          • Assignee:
            shalinmangar Shalin Shekhar Mangar
            Reporter:
            shalinmangar Shalin Shekhar Mangar
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 24h
              24h
              Remaining:
              Remaining Estimate - 24h
              24h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development