Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2988

"Error on initialization of server mk-worker" when using org.apache.storm.metrics2.reporters.JmxStormReporter on worker

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.0, 1.2.1
    • 2.0.0, 1.2.2
    • storm-client, storm-core
    • CentOS 7.4
      java.version=1.8.0_161
      java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre

    Description

      As per documentation, I configured metrics v2 in my storm.yaml using the following configuration:
       

      storm.metrics.reporters: 
      
        - class: "org.apache.storm.metrics2.reporters.JmxStormReporter"
          daemons: 
              - "supervisor"
              - "nimbus"
              - "worker"
          report.period: 10
          report.period.units: "SECONDS"
      

      When I start nimbus and supervisors everything works properly, I can see metrics reported to JMX, and logs (for nimbus in this example) report:

      2018-03-07 15:35:22.201 o.a.s.d.m.MetricsUtils main [INFO] Using statistics reporter plugin:org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter
      2018-03-07 15:35:22.203 o.a.s.d.m.r.JmxPreparableReporter main [INFO] Preparing...
      2018-03-07 15:35:22.221 o.a.s.d.common main [INFO] Started statistics report plugin...
      

      When I submit a topology, workers cannot initialize and report this error

      2018-03-07 15:39:19.136 o.a.s.d.worker main [INFO] Launching worker for stp_topology-1-1520433551 on [... cut ...]
      2018-03-07 15:39:19.169 o.a.s.m.StormMetricRegistry main [INFO] Starting metrics reporters...
      2018-03-07 15:39:19.172 o.a.s.m.StormMetricRegistry main [INFO] Attempting to instantiate reporter class: org.apache.storm.metrics2.reporters.JmxStormReporter
      2018-03-07 15:39:19.175 o.a.s.m.r.JmxStormReporter main [INFO] Preparing...
      2018-03-07 15:39:19.182 o.a.s.d.worker main [ERROR] Error on initialization of server mk-worker
      java.lang.IllegalArgumentException: Don't know how to convert {"class" "org.apache.storm.metrics2.reporters.JmxStormReporter", "daemons" ["supervisor" "nimbus" "worker"], "report.period" 10, "report.period.units" "SECONDS"} + to String
      	at org.apache.storm.utils.Utils.getString(Utils.java:848) ~[storm-core-1.2.1.jar:1.2.1]
      	at org.apache.storm.metrics2.reporters.JmxStormReporter.getMetricsJMXDomain(JmxStormReporter.java:70) ~[storm-core-1.2.1.jar:1.2.1]
      	at org.apache.storm.metrics2.reporters.JmxStormReporter.prepare(JmxStormReporter.java:51) ~[storm-core-1.2.1.jar:1.2.1]
      	at org.apache.storm.metrics2.StormMetricRegistry.startReporter(StormMetricRegistry.java:119) ~[storm-core-1.2.1.jar:1.2.1]
      	at org.apache.storm.metrics2.StormMetricRegistry.start(StormMetricRegistry.java:102) ~[storm-core-1.2.1.jar:1.2.1]
      	at org.apache.storm.daemon.worker$fn__5545$exec_fn__1369__auto____5546.invoke(worker.clj:611) ~[storm-core-1.2.1.jar:1.2.1]
      	at clojure.lang.AFn.applyToHelper(AFn.java:178) ~[clojure-1.7.0.jar:?]
      	at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
      	at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]
      	at org.apache.storm.daemon.worker$fn__5545$mk_worker__5636.doInvoke(worker.clj:598) [storm-core-1.2.1.jar:1.2.1]
      	at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.7.0.jar:?]
      	at org.apache.storm.daemon.worker$_main.invoke(worker.clj:787) [storm-core-1.2.1.jar:1.2.1]
      	at clojure.lang.AFn.applyToHelper(AFn.java:165) [clojure-1.7.0.jar:?]
      	at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.7.0.jar:?]
      	at org.apache.storm.daemon.worker.main(Unknown Source) [storm-core-1.2.1.jar:1.2.1]
      2018-03-07 15:39:19.195 o.a.s.util main [ERROR] Halting process: ("Error on initialization")
      java.lang.RuntimeException: ("Error on initialization")
      	at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.2.1.jar:1.2.1]
      	at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
      	at org.apache.storm.daemon.worker$fn__5545$mk_worker__5636.doInvoke(worker.clj:598) [storm-core-1.2.1.jar:1.2.1]
      	at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.7.0.jar:?]
      	at org.apache.storm.daemon.worker$_main.invoke(worker.clj:787) [storm-core-1.2.1.jar:1.2.1]
      	at clojure.lang.AFn.applyToHelper(AFn.java:165) [clojure-1.7.0.jar:?]
      	at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.7.0.jar:?]
      	at org.apache.storm.daemon.worker.main(Unknown Source) [storm-core-1.2.1.jar:1.2.1]
      

      Looking at org.apache.storm.metrics2.reporters.JmxStormReporter.getMetricsJMXDomain() I found that it passes "reporterConf" map to Utils.getString() instead of a string:

          public static String getMetricsJMXDomain(Map reporterConf) {
              return Utils.getString(reporterConf, JMX_DOMAIN);
      }
      

      The "prepare" method in org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter used by nimbus and supervisor correctly passes a string to Utils.getString():

      public void prepare(MetricRegistry metricsRegistry, Map stormConf) {
              LOG.info("Preparing...");
              JmxReporter.Builder builder = JmxReporter.forRegistry(metricsRegistry);
              String domain = Utils.getString(stormConf.get(Config.STORM_DAEMON_METRICS_REPORTER_PLUGIN_DOMAIN), null);
              if (domain != null) {
                  builder.inDomain(domain);
      }
      [...]
      

      Is this a bug or am I missing something in configuration?

      Regards,
      Federico Chiacchiaretta

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            dbist13 Artem Ervits Assign to me
            fchiacchiaretta Federico Chiacchiaretta
            Votes:
            2 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - Not Specified
              Not Specified
              Remaining:
              Remaining Estimate - 0h
              0h
              Logged:
              Time Spent - 4.5h
              4.5h

              Slack

                Issue deployment