Kafka
  1. Kafka
  2. KAFKA-901

Kafka server can become unavailable if clients send several metadata requests

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: None
    • Component/s: replication
    • Labels:
      None

      Description

      Currently, if a broker is bounced without controlled shutdown and there are several clients talking to the Kafka cluster, each of the clients realize the unavailability of leaders for some partitions. This leads to several metadata requests sent to the Kafka brokers. Since metadata requests are pretty slow, all the I/O threads quickly become busy serving the metadata requests. This leads to a full request queue, that stalls handling of finished responses since the same network thread handles requests as well as responses. In this situation, clients timeout on metadata requests and send more metadata requests. This quickly makes the Kafka cluster unavailable.

      1. metadata-request-improvement.patch
        91 kB
        Neha Narkhede
      2. kafka-901.patch
        117 kB
        Neha Narkhede
      3. kafka-901-v2.patch
        138 kB
        Neha Narkhede
      4. kafka-901-v4.patch
        133 kB
        Neha Narkhede
      5. kafka-901-v5.patch
        134 kB
        Neha Narkhede
      6. kafka-901-followup.patch
        4 kB
        Neha Narkhede
      7. kafka-901-followup2.patch
        5 kB
        Neha Narkhede

        Activity

        Neha Narkhede created issue -
        Neha Narkhede made changes -
        Field Original Value New Value
        Status Open [ 1 ] In Progress [ 3 ]
        Neha Narkhede made changes -
        Status In Progress [ 3 ] Patch Available [ 10002 ]
        Neha Narkhede made changes -
        Attachment metadata-request-improvement.patch [ 12582557 ]
        Neha Narkhede made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Neha Narkhede made changes -
        Attachment kafka-901.patch [ 12583245 ]
        Neha Narkhede made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Neha Narkhede made changes -
        Attachment kafka-901-v2.patch [ 12583495 ]
        Neha Narkhede made changes -
        Status Patch Available [ 10002 ] In Progress [ 3 ]
        Neha Narkhede made changes -
        Status In Progress [ 3 ] Patch Available [ 10002 ]
        Neha Narkhede made changes -
        Attachment kafka-901-v4.patch [ 12583603 ]
        Neha Narkhede made changes -
        Attachment kafka-901-v5.patch [ 12583681 ]
        Neha Narkhede made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Neha Narkhede made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Neha Narkhede made changes -
        Attachment kafka-901-followup.patch [ 12583861 ]
        Neha Narkhede made changes -
        Attachment kafka-901-followup2.patch [ 12584030 ]

          People

          • Assignee:
            Neha Narkhede
            Reporter:
            Neha Narkhede
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development