Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.7
    • Fix Version/s: None
    • Component/s: core
    • Labels:
      None

      Description

      There are two cases where correct partitioning fails for a new topic.

      Case 1: Topic exists on current Kafka cluster. A new broker is added to the cluster. The new broker will never host partitions for the existing topic.

      To reproduce:
      1) Create a cluster of brokers along with a ZooKeeper ensemble.
      2) Send messages for a topic to the cluster.
      3) Add a new broker to the cluster.
      4) New broker will never see the existing topic.

      Case 2: Topic does not exist on current Kafka cluster. Producer sends messages to a new topic that did not previously exist in the cluster. If, during the producer session, one or more partitions are not created on a broker, the broker will never host those partitions.

      To reproduce:
      1) Create a cluster of brokers along with a ZooKeeper ensemble.
      2) Send messages to a new topic.
      3) Shut down the producer before the topic is created on at least one broker.
      4) The broker that did not allocate the topic will never host the topic.

      My guess here is that when a new producer is created, it gets a list of topics and partitions based on the current state of the brokers in the cluster. Since some brokers are missing the topic, the producer will never send messages to that broker and partitions will never be created.

      Work around:
      Manually create the topic/partition directories in the kafka logs directory and reboot kafka. It will register the topic/partitions in ZooKeeper.

        Activity

        Hide
        Matt added a comment -

        I changed the description and title to reflect that there are 2 ways that this condition can arise.

        Show
        Matt added a comment - I changed the description and title to reflect that there are 2 ways that this condition can arise.
        Hide
        Matt added a comment -

        Refined the steps to reproduce.

        Show
        Matt added a comment - Refined the steps to reproduce.
        Hide
        Matt added a comment -

        Added a work around.

        Show
        Matt added a comment - Added a work around.
        Hide
        Nick Howard added a comment -

        Here's a patch we applied to our fork to deal with this issue.

        What it does is bootstrap new brokers in the same way existing brokers are bootstrapped for new topics.

        It includes a test.

        Show
        Nick Howard added a comment - Here's a patch we applied to our fork to deal with this issue. What it does is bootstrap new brokers in the same way existing brokers are bootstrapped for new topics. It includes a test.
        Hide
        Ian Friedman added a comment -

        Hi, I didn't realize it was expected for multiple brokers to split partitions of a single topic in 0.7? Can anyone point me to where that behavior is documented?

        Show
        Ian Friedman added a comment - Hi, I didn't realize it was expected for multiple brokers to split partitions of a single topic in 0.7? Can anyone point me to where that behavior is documented?
        Hide
        Jun Rao added a comment -

        Basically, in 0.7, a partition is local to a broker. So, if you send data to a broker, a partition for that topic will be automatically created in that broker.

        Show
        Jun Rao added a comment - Basically, in 0.7, a partition is local to a broker. So, if you send data to a broker, a partition for that topic will be automatically created in that broker.
        Hide
        Jay Kreps added a comment -

        This should be fixed in 0.8 where we handle partitioning properly.

        Show
        Jay Kreps added a comment - This should be fixed in 0.8 where we handle partitioning properly.

          People

          • Assignee:
            Unassigned
            Reporter:
            Matt
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development