Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-14524

Modularize `core` monolith

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      The `core` module has grown too large and it's time to split it into multiple modules.  The `core` module will be deleted once it's empty (a KIP will be required for this).

      Evidence of `core` growing too large is that it takes 1m10s to compile the main code and tests and it takes hours to run all the tests sequentially.

      As part of this effort, we should rewrite the Scala code in Java to reduce developer friction, reduce compilation time and simplify deployment (i.e. we can remove the scala version suffix from the module name). Scala may have a number of advantages over Java 8 (minimum version we support now) and Java 11 (minimum version we will support in Kafka 4.0), but a mixture of Scala and Java (as we have now) is more complex than just Java.

      Another benefit is that code dependencies will be strictly enforced, which will hopefully help ensure better abstractions.

      This pattern was started with the `tools` (but not completed), `metadata` and `raft` modules and we have (when this ticket was filed) a couple more in progress: `group-coordinator` and `storage`.

      This is an umbrella ticket and it will link to each ticket related to this goal.

      Attachments

        Issue Links

          is blocked by

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-14470 Move log layer to storage module

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-14525 Move CLI tools from `core` to `tools` module

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-15852 Move server code from `core` to `server` module

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-14048 The Next Generation of the Consumer Rebalance Protocol

          • Major - Major loss of function.
          • Reopened

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-14526 Move `kafka.network` from `core` to separate module

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-14527 Move `kafka.security` from `core` to separate module

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. KAFKA-14528 Move replication code from `core` to separate module

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #13067

          Activity

            People

              Unassigned Unassigned
              ijuma Ismael Juma
              Votes:
              1 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated: