[STORM-2820] validateTopologyWorkerMaxHeapSizeConfigs function never picks up the value set by nimbus

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Labels: pull-request-available
• Assignee: Ethan Li
• Reporter: Ethan Li
• Created: 15/Nov/17 20:12
• Updated: 16/Nov/17 23:00
• Resolved: 16/Nov/17 23:00

Description

https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java#L2548

   @VisibleForTesting
    static void validateTopologyWorkerMaxHeapSizeConfigs(
        Map<String, Object> stormConf, StormTopology topology) {
        double largestMemReq = getMaxExecutorMemoryUsageForTopo(topology, stormConf);
        double topologyWorkerMaxHeapSize =
            ObjectReader.getDouble(stormConf.get(Config.TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB), 768.0);
        if (topologyWorkerMaxHeapSize < largestMemReq) {
            throw new IllegalArgumentException(
                "Topology will not be able to be successfully scheduled: Config "
                    + "TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB="
                    + topologyWorkerMaxHeapSize
                    + " < " + largestMemReq + " (Largest memory requirement of a component in the topology)."
                    + " Perhaps set TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB to a larger amount");
        }
    }

The topologyWorkerMaxHeapSize in the above code is either the value from topology configuration (set by the topology) or 768.0. It never picks up the value set by nimbus .

A test:
I set

topology.worker.max.heap.size.mb: 2000.0

in storm.yaml.

And have the WordCountTopology to have the WordCount bolt with memory load of 1024MB.

builder.setBolt("count", new WordCount(), 12).fieldsGrouping("split", new Fields("word")).setMemoryLoad(1024);

I got an error when submitting this topology. The nimbus log shows

2017-11-15 19:46:43.085 o.a.s.d.n.Nimbus pool-14-thread-2 [WARN] Topology submission exception. (topology name='wc')
java.lang.IllegalArgumentException: Topology will not be able to be successfully scheduled: Config TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB=768.0 < 1024.0 (Largest memory requirement of a component in the topology). Perhaps set TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB to a larger amount

Issue Links

• links to: GitHub Pull Request #2424