Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.0
    • Component/s: None
    • Labels:

      Description

      It should be possible to increase the partition count using the AdminClient.

      See KIP-195

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user tombentley opened a pull request:

          https://github.com/apache/kafka/pull/3870

          KAFKA-5856: Add AdminClient.createPartitions()

          See KIP-195.

          The contribution is my original work and I license the work to the project under the project's open source license.

          This patch adds AdminClient.createPartitions() and the network protocol is
          uses. The broker-side algorithm is as follows:

          1. KafkaApis makes some initial checks on the request, then delegates to the
          new AdminManager.createPartitions() method.
          2. AdminManager.createPartitions() performs some validation then delegates to
          AdminUtils.addPartitions().

          Aside: I felt it was safer to add the extra validation in
          AdminManager.createPartitions() than in AdminUtils.addPartitions() since the
          latter is used on other code paths which might fail differently with the
          introduction of extra checks.

          3. AdminUtils.addPartitions() does its own checks and adds the partitions.
          4. AdminManager then uses the existing topic purgatory to wait for the
          PartitionInfo available from the metadata cache to become consistent with
          the new total number of partitions.

          The messages of exceptions thrown in AdminUtils affecting this new API have
          been made consistent with initial capital letter and terminating period.
          A few have been reworded for clarity. I've also standardized on using
          String.format().

          cc @ijuma

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/tombentley/kafka KAFKA-5856-AdminClient.createPartitions

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/kafka/pull/3870.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3870


          commit ab07f15a794c385cbfdecd33a5a44c7725e8d103
          Author: Tom Bentley <tbentley@redhat.com>
          Date: 2017-09-15T09:50:59Z

          KAFKA-5856: Add AdminClient.createPartitions()

          See KIP-195.

          This patch adds AdminClient.createPartitions() and the network protocol is
          uses. The broker-side algorithm is as follows:

          1. KafkaApis makes some initial checks on the request, then delegates to the
          new AdminManager.createPartitions() method.
          2. AdminManager.createPartitions() performs some validation then delegates to
          AdminUtils.addPartitions().

          Aside: I felt it was safer to add the extra validation in
          AdminManager.createPartitions() than in AdminUtils.addPartitions() since the
          latter is used on other code paths which might fail differently with the
          introduction of extra checks.

          3. AdminUtils.addPartitions() does its own checks and adds the partitions.
          4. AdminManager then uses the existing topic purgatory to wait for the
          PartitionInfo available from the metadata cache to become consistent with
          the new total number of partitions.

          The messages of exceptions thrown in AdminUtils affecting this new API have
          been made consistent with initial capital letter and terminating period.
          A few have been reworded for clarity. I've also standardized on using
          String.format().


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user tombentley opened a pull request: https://github.com/apache/kafka/pull/3870 KAFKA-5856 : Add AdminClient.createPartitions() See KIP-195. The contribution is my original work and I license the work to the project under the project's open source license. This patch adds AdminClient.createPartitions() and the network protocol is uses. The broker-side algorithm is as follows: 1. KafkaApis makes some initial checks on the request, then delegates to the new AdminManager.createPartitions() method. 2. AdminManager.createPartitions() performs some validation then delegates to AdminUtils.addPartitions(). Aside: I felt it was safer to add the extra validation in AdminManager.createPartitions() than in AdminUtils.addPartitions() since the latter is used on other code paths which might fail differently with the introduction of extra checks. 3. AdminUtils.addPartitions() does its own checks and adds the partitions. 4. AdminManager then uses the existing topic purgatory to wait for the PartitionInfo available from the metadata cache to become consistent with the new total number of partitions. The messages of exceptions thrown in AdminUtils affecting this new API have been made consistent with initial capital letter and terminating period. A few have been reworded for clarity. I've also standardized on using String.format(). cc @ijuma You can merge this pull request into a Git repository by running: $ git pull https://github.com/tombentley/kafka KAFKA-5856 -AdminClient.createPartitions Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3870.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3870 commit ab07f15a794c385cbfdecd33a5a44c7725e8d103 Author: Tom Bentley <tbentley@redhat.com> Date: 2017-09-15T09:50:59Z KAFKA-5856 : Add AdminClient.createPartitions() See KIP-195. This patch adds AdminClient.createPartitions() and the network protocol is uses. The broker-side algorithm is as follows: 1. KafkaApis makes some initial checks on the request, then delegates to the new AdminManager.createPartitions() method. 2. AdminManager.createPartitions() performs some validation then delegates to AdminUtils.addPartitions(). Aside: I felt it was safer to add the extra validation in AdminManager.createPartitions() than in AdminUtils.addPartitions() since the latter is used on other code paths which might fail differently with the introduction of extra checks. 3. AdminUtils.addPartitions() does its own checks and adds the partitions. 4. AdminManager then uses the existing topic purgatory to wait for the PartitionInfo available from the metadata cache to become consistent with the new total number of partitions. The messages of exceptions thrown in AdminUtils affecting this new API have been made consistent with initial capital letter and terminating period. A few have been reworded for clarity. I've also standardized on using String.format().
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/kafka/pull/3870

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3870
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user ijuma opened a pull request:

          https://github.com/apache/kafka/pull/3930

          KAFKA-5856; AdminClient.createPartitions() follow-up (KIP-195)

          • Remove DelayedCreatePartitions to reduce code duplication
          • Avoid unnecessary ZK calls (call it once per request instead
            of once per topic, if possible)
          • Simplify code
          • A few minor clean-ups

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/ijuma/kafka kafka-5856-admin-client-creation-partitions

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/kafka/pull/3930.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3930



          Show
          githubbot ASF GitHub Bot added a comment - GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/3930 KAFKA-5856 ; AdminClient.createPartitions() follow-up (KIP-195) Remove DelayedCreatePartitions to reduce code duplication Avoid unnecessary ZK calls (call it once per request instead of once per topic, if possible) Simplify code A few minor clean-ups You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijuma/kafka kafka-5856-admin-client-creation-partitions Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3930.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3930

            People

            • Assignee:
              tombentley Tom Bentley
              Reporter:
              tombentley Tom Bentley
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development