Flume
  1. Flume
  2. FLUME-1071

Evaluate performance of AVRO source with multiple clients

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v1.2.0
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Environment:

      CentOS 6.2 64-bit

      Description

      I ran an AVRO stress test with 10 avro clients continuously sending data. Those clients all target the same port.

      A profiling run (pict att'd) shows that the 10 worker threads are waiting a lot (yellow) and little time spent running (green).

      One thing to try for future tests, as a basis for comparison, would be to use an increased heap size because in general flume should give better performance with more mem.

      Note that a source's performance will likely depend on the channel in use (I'm using memory channel in this case), since the transactions are the primary thing that can effect blocking.

        Activity

        Will McQueen created issue -
        Mike Percy made changes -
        Field Original Value New Value
        Fix Version/s v1.3.0 [ 12322140 ]
        Fix Version/s v1.2.0 [ 12320243 ]
        Brock Noland made changes -
        Fix Version/s v1.4.0 [ 12323372 ]
        Fix Version/s v1.3.0 [ 12322140 ]
        Israel Ekpo made changes -
        Issue Type Task [ 3 ] Improvement [ 4 ]
        Mike Percy made changes -
        Fix Version/s v1.4.0 [ 12323372 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Will McQueen
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development