Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-2517

Performance Regression post SSL implementation

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0.0
    • Component/s: None
    • Labels:
      None

      Description

      It would appear that we incurred a performance regression on submission of the SSL work affecting the performance of the new Kafka Consumer.

      Running with 1KB messages. Macbook 2.3 GHz Intel Core i7, 8GB, APPLE SSD SM256E. Single server instance. All local.

      kafka-consumer-perf-test.sh ... --messages 3000000 --new-consumer

      Pre-SSL changes (commit 503bd36647695e8cc91893ffb80346dd03eb0bc5)
      Steady state throughputs = 234.8 MB/s
      (2861.5913, 234.8261, 3000596, 246233.0543)

      Post-SSL changes (commit 13c432f7952de27e9bf8cb4adb33a91ae3a4b738)
      Steady state throughput = 178.1 MB/s
      (2861.5913, 178.1480, 3000596, 186801.7182)

      Implication is a 25% reduction in consumer throughput for these test conditions.

      This appears to be caused by the use of PlaintextTransportLayer rather than SocketChannel in FileMessageSet.writeTo() meaning a zero copy transfer is not invoked.

      Switching to the use of a SocketChannel directly in FileMessageSet.writeTo() yields the following result:
      Steady state throughput = 281.8 MB/s
      (2861.5913, 281.8191, 3000596, 295508.7650)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ijuma Ismael Juma
                Reporter:
                benstopford Ben Stopford
                Reviewer:
                Jun Rao
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: