[STORM-2988] "Error on initialization of server mk-worker" when using org.apache.storm.metrics2.reporters.JmxStormReporter on worker

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0, 1.2.1
• Fix Version/s: 2.0.0, 1.2.2
• Component/s: storm-client, storm-core
• Labels: pull-request-available
• Environment: CentOS 7.4 ; java.version=1.8.0_161 ; java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre
• Assignee: Artem Ervits
• Reporter: Federico Chiacchiaretta
• Created: 07/Mar/18 14:55
• Updated: 21/May/18 18:11
• Resolved: 14/May/18 04:39

Description

As per documentation, I configured metrics v2 in my storm.yaml using the following configuration:
 

storm.metrics.reporters: 

  - class: "org.apache.storm.metrics2.reporters.JmxStormReporter"
    daemons: 
        - "supervisor"
        - "nimbus"
        - "worker"
    report.period: 10
    report.period.units: "SECONDS"

When I start nimbus and supervisors everything works properly, I can see metrics reported to JMX, and logs (for nimbus in this example) report:

2018-03-07 15:35:22.201 o.a.s.d.m.MetricsUtils main [INFO] Using statistics reporter plugin:org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter
2018-03-07 15:35:22.203 o.a.s.d.m.r.JmxPreparableReporter main [INFO] Preparing...
2018-03-07 15:35:22.221 o.a.s.d.common main [INFO] Started statistics report plugin...

When I submit a topology, workers cannot initialize and report this error

2018-03-07 15:39:19.136 o.a.s.d.worker main [INFO] Launching worker for stp_topology-1-1520433551 on [... cut ...]
2018-03-07 15:39:19.169 o.a.s.m.StormMetricRegistry main [INFO] Starting metrics reporters...
2018-03-07 15:39:19.172 o.a.s.m.StormMetricRegistry main [INFO] Attempting to instantiate reporter class: org.apache.storm.metrics2.reporters.JmxStormReporter
2018-03-07 15:39:19.175 o.a.s.m.r.JmxStormReporter main [INFO] Preparing...
2018-03-07 15:39:19.182 o.a.s.d.worker main [ERROR] Error on initialization of server mk-worker
java.lang.IllegalArgumentException: Don't know how to convert {"class" "org.apache.storm.metrics2.reporters.JmxStormReporter", "daemons" ["supervisor" "nimbus" "worker"], "report.period" 10, "report.period.units" "SECONDS"} + to String
	at org.apache.storm.utils.Utils.getString(Utils.java:848) ~[storm-core-1.2.1.jar:1.2.1]
	at org.apache.storm.metrics2.reporters.JmxStormReporter.getMetricsJMXDomain(JmxStormReporter.java:70) ~[storm-core-1.2.1.jar:1.2.1]
	at org.apache.storm.metrics2.reporters.JmxStormReporter.prepare(JmxStormReporter.java:51) ~[storm-core-1.2.1.jar:1.2.1]
	at org.apache.storm.metrics2.StormMetricRegistry.startReporter(StormMetricRegistry.java:119) ~[storm-core-1.2.1.jar:1.2.1]
	at org.apache.storm.metrics2.StormMetricRegistry.start(StormMetricRegistry.java:102) ~[storm-core-1.2.1.jar:1.2.1]
	at org.apache.storm.daemon.worker$fn__5545$exec_fn__1369__auto____5546.invoke(worker.clj:611) ~[storm-core-1.2.1.jar:1.2.1]
	at clojure.lang.AFn.applyToHelper(AFn.java:178) ~[clojure-1.7.0.jar:?]
	at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
	at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]
	at org.apache.storm.daemon.worker$fn__5545$mk_worker__5636.doInvoke(worker.clj:598) [storm-core-1.2.1.jar:1.2.1]
	at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.7.0.jar:?]
	at org.apache.storm.daemon.worker$_main.invoke(worker.clj:787) [storm-core-1.2.1.jar:1.2.1]
	at clojure.lang.AFn.applyToHelper(AFn.java:165) [clojure-1.7.0.jar:?]
	at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.7.0.jar:?]
	at org.apache.storm.daemon.worker.main(Unknown Source) [storm-core-1.2.1.jar:1.2.1]
2018-03-07 15:39:19.195 o.a.s.util main [ERROR] Halting process: ("Error on initialization")
java.lang.RuntimeException: ("Error on initialization")
	at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.2.1.jar:1.2.1]
	at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
	at org.apache.storm.daemon.worker$fn__5545$mk_worker__5636.doInvoke(worker.clj:598) [storm-core-1.2.1.jar:1.2.1]
	at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.7.0.jar:?]
	at org.apache.storm.daemon.worker$_main.invoke(worker.clj:787) [storm-core-1.2.1.jar:1.2.1]
	at clojure.lang.AFn.applyToHelper(AFn.java:165) [clojure-1.7.0.jar:?]
	at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.7.0.jar:?]
	at org.apache.storm.daemon.worker.main(Unknown Source) [storm-core-1.2.1.jar:1.2.1]

Looking at org.apache.storm.metrics2.reporters.JmxStormReporter.getMetricsJMXDomain() I found that it passes "reporterConf" map to Utils.getString() instead of a string:

    public static String getMetricsJMXDomain(Map reporterConf) {
        return Utils.getString(reporterConf, JMX_DOMAIN);
}

The "prepare" method in org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter used by nimbus and supervisor correctly passes a string to Utils.getString():

public void prepare(MetricRegistry metricsRegistry, Map stormConf) {
        LOG.info("Preparing...");
        JmxReporter.Builder builder = JmxReporter.forRegistry(metricsRegistry);
        String domain = Utils.getString(stormConf.get(Config.STORM_DAEMON_METRICS_REPORTER_PLUGIN_DOMAIN), null);
        if (domain != null) {
            builder.inDomain(domain);
}
[...]

Is this a bug or am I missing something in configuration?

Regards,
Federico Chiacchiaretta

Issue Links

• relates to: STORM-3062 Document JMXStormReporter configuration
• links to: GitHub Pull Request #2665
• links to: GitHub Pull Request #2666