Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2820

validateTopologyWorkerMaxHeapSizeConfigs function never picks up the value set by nimbus

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: None

      Description

      https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java#L2548

         @VisibleForTesting
          static void validateTopologyWorkerMaxHeapSizeConfigs(
              Map<String, Object> stormConf, StormTopology topology) {
              double largestMemReq = getMaxExecutorMemoryUsageForTopo(topology, stormConf);
              double topologyWorkerMaxHeapSize =
                  ObjectReader.getDouble(stormConf.get(Config.TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB), 768.0);
              if (topologyWorkerMaxHeapSize < largestMemReq) {
                  throw new IllegalArgumentException(
                      "Topology will not be able to be successfully scheduled: Config "
                          + "TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB="
                          + topologyWorkerMaxHeapSize
                          + " < " + largestMemReq + " (Largest memory requirement of a component in the topology)."
                          + " Perhaps set TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB to a larger amount");
              }
          }
      

      The topologyWorkerMaxHeapSize in the above code is either the value from topology configuration (set by the topology) or 768.0. It never picks up the value set by nimbus .

      A test:
      I set

      topology.worker.max.heap.size.mb: 2000.0
      

      in storm.yaml.

      And have the WordCountTopology to have the WordCount bolt with memory load of 1024MB.

      builder.setBolt("count", new WordCount(), 12).fieldsGrouping("split", new Fields("word")).setMemoryLoad(1024);
      

      I got an error when submitting this topology. The nimbus log shows

      2017-11-15 19:46:43.085 o.a.s.d.n.Nimbus pool-14-thread-2 [WARN] Topology submission exception. (topology name='wc')
      java.lang.IllegalArgumentException: Topology will not be able to be successfully scheduled: Config TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB=768.0 < 1024.0 (Largest memory requirement of a component in the topology). Perhaps set TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB to a larger amount
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ethanli Ethan Li
                Reporter:
                ethanli Ethan Li
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m