Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-1409

oversized messages can slow down the brokers

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.8.2.0
    • Fix Version/s: 0.8.2.0
    • Component/s: core
    • Labels:
      None

      Description

      There are 2 main issues.

      1. The broker first decompresses and then recompresses each message (to assign new offsets) before validating the message size (we have to do the validation after recompression since the message size could change). So, it can spend many secs to decompress/recompress an oversized message, only to be dropped later. While this was happening, a request thread was tied up, which reduced the capacity on the broker.

      2. Both the fetch and producer requests need to hold a leader lock (per partition). So, if the producer is slow in appending the log, it will block other producer/fetch requests on the same partition.

        Attachments

        1. KAFKA-1409.patch
          18 kB
          Guozhang Wang
        2. KAFKA-1409_2014-04-25_10:44:27.patch
          17 kB
          Guozhang Wang
        3. KAFKA-1409_2014-04-25_10:22:54.patch
          17 kB
          Guozhang Wang
        4. KAFKA-1409_2014-04-24_16:26:06.patch
          17 kB
          Guozhang Wang
        5. KAFKA-1409_2014-04-23_15:20:43.patch
          17 kB
          Guozhang Wang
        6. KAFKA-1409_2014-04-23_14:25:48.patch
          18 kB
          Guozhang Wang
        7. KAFKA-1409_2014-04-23_14:10:03.patch
          18 kB
          Guozhang Wang

          Activity

            People

            • Assignee:
              guozhang Guozhang Wang
              Reporter:
              junrao Jun Rao
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 48h
                48h
                Remaining:
                Remaining Estimate - 48h
                48h
                Logged:
                Time Spent - Not Specified
                Not Specified