Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-8629

Kafka Streams Apps to support small native images through GraalVM

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.3.0
    • Fix Version/s: None
    • Component/s: streams
    • Labels:
      None
    • Environment:
      OSX
      Linux on Docker

      Description

      I'm investigating using GraalVM to help with reducing docker image size and required resources for a simple Kafka Streams microservice. To this end, I'm looking at running a microservice which:
      1) consumes from a Kafka topic (XML)
      2) Transforms into JSON
      3) Produces to a new Kafka topic.

      The Kafka Streams app running in the JVM works fine.

      When I attempt to build it to a GraalVM native image (binary executable which does not require the JVM, hence smaller image size and less resources), I encountered a few incompatibilities with the source code in Kafka.

      I've implemented a workaround for each of these in a fork (link to follow) to help establish if it is feasible. I don't intend (at this stage) for the changes to be applied to the broker - I'm only after Kafka Streams for now. I'm not sure whether it'd be a good idea for the broker itself to run as a native image!

      There were 2 issues getting the native image with kafka streams:
      1) Some Reflection use cases using MethodHandle
      2) Anything JMX

      To work around these issues, I have:
      1) Replaced use of MethodHandle with alternatives
      2) Commented out the JMX code in a few places

      While the first may be sustainable, I'd expect that the 2nd option should be put behind a configuration switch to allow the existing code to be used by default and turning off JMX if configured.

      I haven't created a PR for now, as I'd like feedback to decide if it is going to be feasible to carry this forwards.

        Attachments

          Activity

            People

            • Assignee:
              muirandy Andy Muir
              Reporter:
              muirandy Andy Muir
            • Votes:
              4 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated: