[STORM-3667] Exception message on submitting topology due to incorrect worker max heap size should be more straightforward

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Assignee: Ethan Li
• Reporter: Ethan Li
• Created: 06/Jul/20 21:52
• Updated: 15/Jul/20 14:17
• Resolved: 15/Jul/20 14:17

Description

Submitting a topology can fail when the largest component heap size is larger than

topology.worker.max.heap.size.mb

For example, when you submit a topology with the following command

  bin/storm jar /home/y/lib64/jars/storm-starter.jar  org.apache.storm.starter.WordCountTopology -c topology.worker.max.heap.size.mb=100  -c topology.component.resources.onheap.memory.mb=120 wc

While the nimbus log shows

 2020-07-06 21:51:12.026 o.a.s.d.n.Nimbus pool-29-thread-45 [WARN] Topology submission exception. (topology name='wc')
java.lang.IllegalArgumentException: Topology will not be able to be successfully scheduled: Config TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB=100.0 < 120.0 (Largest memory requirement of a component in the topology). Perhaps set TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB to a larger amount
        at org.apache.storm.daemon.nimbus.Nimbus.validateTopologyWorkerMaxHeapSizeConfigs(Nimbus.java:1650) ~[storm-server-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.apache.storm.daemon.nimbus.Nimbus.submitTopologyWithOpts(Nimbus.java:3145) [storm-server-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
        at org.apache.storm.daemon.nimbus.Nimbus.submitTopology(Nimbus.java:3107) [storm-server-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]

the exception returned to users is

 21:51:12.012 [main] INFO  o.a.s.StormSubmitter - Submitting topology wc in distributed mode with conf {"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-8967725949793348049:-5020411463316947973","topology.workers":3,"topology.component.resources.onheap.memory.mb":120,"topology.worker.max.heap.size.mb":100,"topology.debug":true}
java.lang.RuntimeException: org.apache.storm.thrift.TApplicationException: Internal error processing submitTopology
	at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:291)
	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:210)
	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:173)
	at org.apache.storm.topology.ConfigurableTopology.submit(ConfigurableTopology.java:119)
	at org.apache.storm.starter.WordCountTopology.run(WordCountTopology.java:58)
	at org.apache.storm.topology.ConfigurableTopology.start(ConfigurableTopology.java:68)
	at org.apache.storm.starter.WordCountTopology.main(WordCountTopology.java:36)
Caused by: org.apache.storm.thrift.TApplicationException: Internal error processing submitTopology
	at org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
	at org.apache.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:319)
	at org.apache.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:303)
	at org.apache.storm.StormSubmitter.submitTopologyInDistributeMode(StormSubmitter.java:344)
	at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:279)
	... 6 more

which is not very helpful.