Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
-
None
Description
Rather than hand-writing the code for sending and receiving all the different versions of the Kafka RPC protocol, we should have a protocol generator which can generate this code from an interface definition language (IDL).
This will make it less labor-intensive and error-prone to add new message types and versions. It will also make it easier to support these new RPC changes in clients such as librdkafka.
Eventually, we should be able to get rid of the Struct classes and serialize messages directly to byte buffers. This will greatly reduce the garbage collection overhead of the network stack, especially when handling large messages.
Furthermore, having a formal definition for the Kafka protocol may eventually allow us to expose other transports such as REST without writing lots of glue code to translate back and forth.