Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-297

Storm Performance cannot be scaled up by adding more CPU cores

    XMLWordPrintableJSON

Details

    Description

      We cannot scale up the performance by adding more CPU cores and increasing parallelism.

      For a 2 layer topology Spout --shuffle grouping-> bolt, when message size is small (around 100 bytes), we can find in the below picture that neither the CPU nor the network is saturated. When message size is 100 bytes, only 40% of CPU is used, only 18% of network is used, although we have a high parallelism (overall we have 144 executors)

      Attachments

        1. Storm_performance_fix.pdf
          514 kB
          Sean Zhong
        2. storm_performance_fix.patch
          127 kB
          Sean Zhong
        3. storm_Netty_receiver_diagram.png
          65 kB
          Sean Zhong
        4. worker_throughput_without_storm-297.png
          27 kB
          Sean Zhong
        5. storm_conf.txt
          5 kB
          Sean Zhong

        Issue Links

          duplicates

          Sub-task - The sub-task of the issue STORM-271 Naming storm threads more meaningful names.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Sub-task - The sub-task of the issue STORM-272 Make worker receiver thread number configurable

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. STORM-329 Fix cascading Storm failure by improving reconnection strategy and buffering messages

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              clockfly Sean Zhong
              clockfly Sean Zhong
              Votes:
              3 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: