Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-1526

Improve Storm core performance

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.0
    • Component/s: None
    • Labels:
      None

      Description

      Profiling a Speed of Light toplogy running on Storm core without ACKers is showing:

      • Call tree info : shows that a big part of the nextTuple() invocation is consumed in the SpoutOutputCollector.emit() call. 20% of it goes in Reflection by the clojure code

      Method Stats view : Shows that a lot of time is spent blocking on the disruptor queue

      The performance issue is narrowed down to this Clojure code in executor.clj :

      (defn mk-custom-grouper
        [^CustomStreamGrouping grouping ^WorkerTopologyContext context ^String
      component-id ^String stream-id target-tasks]
        (.prepare grouping context (GlobalStreamId. component-id stream-id)
      target-tasks)
        (if (instance? LoadAwareCustomStreamGrouping grouping)
           (fn. [task-id ^List values load]
              (.chooseTasks grouping task-id values load))    ; <-- problematic
      invocation
           (fn [task-id ^List values load]
              (.chooseTasks grouping task-id values))))
      
      

      grouping is statically typed to the base type CustomStreamGrouping. In
      this run, its actual type is the derived type
      LoadAwareCustomStreamGrouping.
      The base type does not have a chooseTasks() method with 3 args. Only the
      derived type has that method. Consequently clojure falls back to
      dynamically iterating over the methods in the grouping object to locate
      the right method & then invoke it appropriately. This falls in the
      critical path SpoutOutputCollector.emit() where it takes about ~20% time
      .. just locating the right method.

        Attachments

          Activity

            People

            • Assignee:
              roshan_naik Roshan Naik
              Reporter:
              roshan_naik Roshan Naik
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: