Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-3667

Exception message on submitting topology due to incorrect worker max heap size should be more straightforward

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.3.0
    • Component/s: None
    • Labels:
      None

      Description

      Submitting a topology can fail when the largest component heap size is larger than

      topology.worker.max.heap.size.mb
      

      For example, when you submit a topology with the following command

        bin/storm jar /home/y/lib64/jars/storm-starter.jar  org.apache.storm.starter.WordCountTopology -c topology.worker.max.heap.size.mb=100  -c topology.component.resources.onheap.memory.mb=120 wc
      

      While the nimbus log shows

       2020-07-06 21:51:12.026 o.a.s.d.n.Nimbus pool-29-thread-45 [WARN] Topology submission exception. (topology name='wc')
      java.lang.IllegalArgumentException: Topology will not be able to be successfully scheduled: Config TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB=100.0 < 120.0 (Largest memory requirement of a component in the topology). Perhaps set TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB to a larger amount
              at org.apache.storm.daemon.nimbus.Nimbus.validateTopologyWorkerMaxHeapSizeConfigs(Nimbus.java:1650) ~[storm-server-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.daemon.nimbus.Nimbus.submitTopologyWithOpts(Nimbus.java:3145) [storm-server-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
              at org.apache.storm.daemon.nimbus.Nimbus.submitTopology(Nimbus.java:3107) [storm-server-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
      

      the exception returned to users is

       21:51:12.012 [main] INFO  o.a.s.StormSubmitter - Submitting topology wc in distributed mode with conf {"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-8967725949793348049:-5020411463316947973","topology.workers":3,"topology.component.resources.onheap.memory.mb":120,"topology.worker.max.heap.size.mb":100,"topology.debug":true}
      java.lang.RuntimeException: org.apache.storm.thrift.TApplicationException: Internal error processing submitTopology
      	at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:291)
      	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:210)
      	at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:173)
      	at org.apache.storm.topology.ConfigurableTopology.submit(ConfigurableTopology.java:119)
      	at org.apache.storm.starter.WordCountTopology.run(WordCountTopology.java:58)
      	at org.apache.storm.topology.ConfigurableTopology.start(ConfigurableTopology.java:68)
      	at org.apache.storm.starter.WordCountTopology.main(WordCountTopology.java:36)
      Caused by: org.apache.storm.thrift.TApplicationException: Internal error processing submitTopology
      	at org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
      	at org.apache.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:319)
      	at org.apache.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:303)
      	at org.apache.storm.StormSubmitter.submitTopologyInDistributeMode(StormSubmitter.java:344)
      	at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:279)
      	... 6 more
      

      which is not very helpful.

        Attachments

          Activity

            People

            • Assignee:
              ethanli Ethan Li
              Reporter:
              ethanli Ethan Li
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 50m
                50m