Kafka
  1. Kafka
  2. KAFKA-1111

Broker prematurely accepts TopicMetadataRequests on startup

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I have an issue where on startup, the broker starts accepting TopicMetadataRequests before it has had metadata sync'd from the controller. This results in a bunch of log entries that look like this:

      013-11-01 03:26:01,577 INFO [kafka-request-handler-0] admin.AdminUtils$ - Topic creation { "partitions":

      { "0":[ 9, 10 ] }

      , "version":1 }
      2013-11-01 03:26:07,767 INFO [kafka-request-handler-1] admin.AdminUtils$ - Topic creation { "partitions":

      { "0":[ 9, 11 ] }

      , "version":1 }
      2013-11-01 03:26:07,823 INFO [kafka-request-handler-1] admin.AdminUtils$ - Topic creation { "partitions":

      { "0":[ 10, 11 ] }

      , "version":1 }
      2013-11-01 03:26:11,183 INFO [kafka-request-handler-2] admin.AdminUtils$ - Topic creation { "partitions":

      { "0":[ 10, 11 ] }

      , "version":1 }

      From an email thread, Neha remarks:

      Before a broker receives the first
      LeaderAndIsrRequest/UpdateMetadataRequest, it is technically not ready to
      start serving any request. But it still ends up serving
      TopicMetadataRequest which can re-create topics accidentally. It shouldn't
      succeed, but this is still a problem.

        Activity

        Hide
        Jason Rosenberg added a comment -

        I've now removed the broker from the metatdata vip rotatation, for a delay period (e.g. 20 seconds), and this does seem to help reduce the preponderance of these Topic creation log messages.

        However, there are still some of these coming through occasionally. It seems this happens when there is a fetch request coming from a consumer. Which I can't block (since consumers don't use the metadata request path, they just use zookeeper). So, I'm guessing this would also be fixed by not allowing the server to start until it's made an attempt to sync up it's metadata.

        Here's an email from the users forum:

        me: Could a fetch request from a consumer cause a Topic creation request
        Neha: Yes, that seems like a way the broker can get into this situation.

        Show
        Jason Rosenberg added a comment - I've now removed the broker from the metatdata vip rotatation, for a delay period (e.g. 20 seconds), and this does seem to help reduce the preponderance of these Topic creation log messages. However, there are still some of these coming through occasionally. It seems this happens when there is a fetch request coming from a consumer. Which I can't block (since consumers don't use the metadata request path, they just use zookeeper). So, I'm guessing this would also be fixed by not allowing the server to start until it's made an attempt to sync up it's metadata. Here's an email from the users forum: me: Could a fetch request from a consumer cause a Topic creation request Neha: Yes, that seems like a way the broker can get into this situation.

          People

          • Assignee:
            Neha Narkhede
            Reporter:
            Jason Rosenberg
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development