[SAMZA-220] SystemConsumers is slow when consuming from a large number of partitions

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.6.0
• Fix Version/s: 0.7.0, 0.8.0
• Component/s: container
• Assignee: Chris Riccomini
• Reporter: Chris Riccomini
• Created: 04/Apr/14 00:40
• Updated: 22/Apr/14 21:20
• Resolved: 22/Apr/14 21:20

Description

We have observed poor throughput when a SamzaContainer is consuming many partitions (100s). The more partitions, the worse the performance gets.

When hooking up VisualVM, two operations take up more than 65% of the CPU in SystemConsumers:

    refresh.maybeCall()
    updateMessageChooser

The problem is that we run each of these operations once before every process() call to a StreamTask. Both of these operations iterate over all SystemStreamPartitions that the SystemConsumers is consuming from. If you have hundreds of partitions, it means you do two loops of 100+ items for every message you process. This is true even if the SystemConsumers buffer has a lot of messages (10,000+), and also true even if most systemStreamPartitions have no messages available.

I have two proposed solutions to this problem:

1. Only call refresh.maybeCall() when the total number of buffered messages in the SystemConsumers has dropped below some low watermark.
2. Only have updateMessageChooser call messageChooser.update for systemStreamPartitions that actually have a message.

I have implemented this and deployed it on a few jobs, and I am seeing significant performance improvement. From 10k-20k msgs/sec to 50k+.

The trade off, as I see it is really around (1), which will introduce a little latency for topics that are low volume. In such a case, the time from when a message arrives to when it gets refreshed in the buffer, and updated in the chooser increases.

Issue Links

• is related to: SAMZA-245 Improve SystemConsumers performance