Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2988

"Error on initialization of server mk-worker" when using org.apache.storm.metrics2.reporters.JmxStormReporter on worker

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0, 1.2.1
    • Fix Version/s: 2.0.0, 1.2.2
    • Component/s: storm-client, storm-core
    • Environment:
      CentOS 7.4
      java.version=1.8.0_161
      java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre

      Description

      As per documentation, I configured metrics v2 in my storm.yaml using the following configuration:
       

      storm.metrics.reporters: 
      
        - class: "org.apache.storm.metrics2.reporters.JmxStormReporter"
          daemons: 
              - "supervisor"
              - "nimbus"
              - "worker"
          report.period: 10
          report.period.units: "SECONDS"
      

      When I start nimbus and supervisors everything works properly, I can see metrics reported to JMX, and logs (for nimbus in this example) report:

      2018-03-07 15:35:22.201 o.a.s.d.m.MetricsUtils main [INFO] Using statistics reporter plugin:org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter
      2018-03-07 15:35:22.203 o.a.s.d.m.r.JmxPreparableReporter main [INFO] Preparing...
      2018-03-07 15:35:22.221 o.a.s.d.common main [INFO] Started statistics report plugin...
      

      When I submit a topology, workers cannot initialize and report this error

      2018-03-07 15:39:19.136 o.a.s.d.worker main [INFO] Launching worker for stp_topology-1-1520433551 on [... cut ...]
      2018-03-07 15:39:19.169 o.a.s.m.StormMetricRegistry main [INFO] Starting metrics reporters...
      2018-03-07 15:39:19.172 o.a.s.m.StormMetricRegistry main [INFO] Attempting to instantiate reporter class: org.apache.storm.metrics2.reporters.JmxStormReporter
      2018-03-07 15:39:19.175 o.a.s.m.r.JmxStormReporter main [INFO] Preparing...
      2018-03-07 15:39:19.182 o.a.s.d.worker main [ERROR] Error on initialization of server mk-worker
      java.lang.IllegalArgumentException: Don't know how to convert {"class" "org.apache.storm.metrics2.reporters.JmxStormReporter", "daemons" ["supervisor" "nimbus" "worker"], "report.period" 10, "report.period.units" "SECONDS"} + to String
      	at org.apache.storm.utils.Utils.getString(Utils.java:848) ~[storm-core-1.2.1.jar:1.2.1]
      	at org.apache.storm.metrics2.reporters.JmxStormReporter.getMetricsJMXDomain(JmxStormReporter.java:70) ~[storm-core-1.2.1.jar:1.2.1]
      	at org.apache.storm.metrics2.reporters.JmxStormReporter.prepare(JmxStormReporter.java:51) ~[storm-core-1.2.1.jar:1.2.1]
      	at org.apache.storm.metrics2.StormMetricRegistry.startReporter(StormMetricRegistry.java:119) ~[storm-core-1.2.1.jar:1.2.1]
      	at org.apache.storm.metrics2.StormMetricRegistry.start(StormMetricRegistry.java:102) ~[storm-core-1.2.1.jar:1.2.1]
      	at org.apache.storm.daemon.worker$fn__5545$exec_fn__1369__auto____5546.invoke(worker.clj:611) ~[storm-core-1.2.1.jar:1.2.1]
      	at clojure.lang.AFn.applyToHelper(AFn.java:178) ~[clojure-1.7.0.jar:?]
      	at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
      	at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]
      	at org.apache.storm.daemon.worker$fn__5545$mk_worker__5636.doInvoke(worker.clj:598) [storm-core-1.2.1.jar:1.2.1]
      	at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.7.0.jar:?]
      	at org.apache.storm.daemon.worker$_main.invoke(worker.clj:787) [storm-core-1.2.1.jar:1.2.1]
      	at clojure.lang.AFn.applyToHelper(AFn.java:165) [clojure-1.7.0.jar:?]
      	at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.7.0.jar:?]
      	at org.apache.storm.daemon.worker.main(Unknown Source) [storm-core-1.2.1.jar:1.2.1]
      2018-03-07 15:39:19.195 o.a.s.util main [ERROR] Halting process: ("Error on initialization")
      java.lang.RuntimeException: ("Error on initialization")
      	at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.2.1.jar:1.2.1]
      	at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
      	at org.apache.storm.daemon.worker$fn__5545$mk_worker__5636.doInvoke(worker.clj:598) [storm-core-1.2.1.jar:1.2.1]
      	at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.7.0.jar:?]
      	at org.apache.storm.daemon.worker$_main.invoke(worker.clj:787) [storm-core-1.2.1.jar:1.2.1]
      	at clojure.lang.AFn.applyToHelper(AFn.java:165) [clojure-1.7.0.jar:?]
      	at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.7.0.jar:?]
      	at org.apache.storm.daemon.worker.main(Unknown Source) [storm-core-1.2.1.jar:1.2.1]
      

      Looking at org.apache.storm.metrics2.reporters.JmxStormReporter.getMetricsJMXDomain() I found that it passes "reporterConf" map to Utils.getString() instead of a string:

          public static String getMetricsJMXDomain(Map reporterConf) {
              return Utils.getString(reporterConf, JMX_DOMAIN);
      }
      

      The "prepare" method in org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter used by nimbus and supervisor correctly passes a string to Utils.getString():

      public void prepare(MetricRegistry metricsRegistry, Map stormConf) {
              LOG.info("Preparing...");
              JmxReporter.Builder builder = JmxReporter.forRegistry(metricsRegistry);
              String domain = Utils.getString(stormConf.get(Config.STORM_DAEMON_METRICS_REPORTER_PLUGIN_DOMAIN), null);
              if (domain != null) {
                  builder.inDomain(domain);
      }
      [...]
      

      Is this a bug or am I missing something in configuration?

      Regards,
      Federico Chiacchiaretta

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                dbist13 Artem Ervits
                Reporter:
                fchiacchiaretta Federico Chiacchiaretta
              • Votes:
                2 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 4.5h
                  4.5h