[FLUME-1071] Evaluate performance of AVRO source with multiple clients

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.2.0
• Component/s: Sinks+Sources

Environment

CentOS 6.2 64-bit

• Assignee: Unassigned
• Reporter: Will McQueen
• Created: 27/Mar/12 21:44
• Updated: 25/Jun/13 00:12

Description

I ran an AVRO stress test with 10 avro clients continuously sending data. Those clients all target the same port.

A profiling run (pict att'd) shows that the 10 worker threads are waiting a lot (yellow) and little time spent running (green).

One thing to try for future tests, as a basis for comparison, would be to use an increased heap size because in general flume should give better performance with more mem.

Note that a source's performance will likely depend on the channel in use (I'm using memory channel in this case), since the transactions are the primary thing that can effect blocking.