Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-456

ProducerSendThread calls ListBuffer.size a whole bunch. That is a O(n) operation

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: core
    • Labels:
    • Environment:
      NA

      Description

      Hi all,

      So there are various statements throughout the async code that call 'events.size', mostly for debugging purposes.
      Problem is that this call is O, so it could add up if the batch size is high. (it's a ListBuffer)

      I see this in at least ProducerSendThread (x4), likely more. Will factor this out myself soon when I start hacking on the project, just wanted to put this somewhere.

      1. KAFKA-456.patch
        0.8 kB
        David Arthur

        Activity

        Hide
        mumrah David Arthur added a comment -

        I looked around in other parts of the code for similar situations (lots of calls to ListBuffer#size), didn't see any that were unnecessary.

        In the worst case, this saves two calls to size(); in the common case, it saves one call.

        Show
        mumrah David Arthur added a comment - I looked around in other parts of the code for similar situations (lots of calls to ListBuffer#size), didn't see any that were unnecessary. In the worst case, this saves two calls to size(); in the common case, it saves one call.
        Hide
        mumrah David Arthur added a comment -

        Only compatible with 0.8

        Show
        mumrah David Arthur added a comment - Only compatible with 0.8
        Hide
        junrao Jun Rao added a comment -

        Thanks for the patch. Committed to 0.8.

        This performance issue seems to have been fixed in scala 2.9.1.
        https://issues.scala-lang.org/browse/SI-4933

        Show
        junrao Jun Rao added a comment - Thanks for the patch. Committed to 0.8. This performance issue seems to have been fixed in scala 2.9.1. https://issues.scala-lang.org/browse/SI-4933
        Hide
        jkreps Jay Kreps added a comment -

        Why not just make this an ArrayBuffer so that there are 0 O operations? I suspect the ListBuffer thing was just a misunderstanding of scala collections underlying implementation (i.e. ListBuffer != ArrayList).

        Show
        jkreps Jay Kreps added a comment - Why not just make this an ArrayBuffer so that there are 0 O operations? I suspect the ListBuffer thing was just a misunderstanding of scala collections underlying implementation (i.e. ListBuffer != ArrayList).

          People

          • Assignee:
            mumrah David Arthur
            Reporter:
            rathboma Matthew Rathbone
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 2h
              2h
              Remaining:
              Remaining Estimate - 2h
              2h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development