Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-220

SystemConsumers is slow when consuming from a large number of partitions

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.7.0, 0.8.0
    • Component/s: container
    • Labels:
      None

      Description

      We have observed poor throughput when a SamzaContainer is consuming many partitions (100s). The more partitions, the worse the performance gets.

      When hooking up VisualVM, two operations take up more than 65% of the CPU in SystemConsumers:

          refresh.maybeCall()
          updateMessageChooser
      

      The problem is that we run each of these operations once before every process() call to a StreamTask. Both of these operations iterate over all SystemStreamPartitions that the SystemConsumers is consuming from. If you have hundreds of partitions, it means you do two loops of 100+ items for every message you process. This is true even if the SystemConsumers buffer has a lot of messages (10,000+), and also true even if most systemStreamPartitions have no messages available.

      I have two proposed solutions to this problem:

      1. Only call refresh.maybeCall() when the total number of buffered messages in the SystemConsumers has dropped below some low watermark.
      2. Only have updateMessageChooser call messageChooser.update for systemStreamPartitions that actually have a message.

      I have implemented this and deployed it on a few jobs, and I am seeing significant performance improvement. From 10k-20k msgs/sec to 50k+.

      The trade off, as I see it is really around (1), which will introduce a little latency for topics that are low volume. In such a case, the time from when a message arrives to when it gets refreshed in the buffer, and updated in the chooser increases.

        Attachments

        1. SAMZA-220.0.patch
          6 kB
          Chris Riccomini
        2. SAMZA-220.1.patch
          13 kB
          Chris Riccomini
        3. SAMZA-220.2.patch
          26 kB
          Chris Riccomini
        4. SAMZA-220.3.patch
          20 kB
          Chris Riccomini
        5. SAMZA-220.4.patch
          22 kB
          Chris Riccomini
        6. without-SAMZA-220.2.png
          222 kB
          Chris Riccomini
        7. with-SAMZA-220.2.png
          220 kB
          Chris Riccomini

          Issue Links

            Activity

              People

              • Assignee:
                criccomini Chris Riccomini
                Reporter:
                criccomini Chris Riccomini
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: