Details
-
New Feature
-
Status: Closed
-
Blocker
-
Resolution: Fixed
-
statefun-2.0.0
Description
For a full multi-language story in Stateful Functions, we're still lacking a YAML-ized way to bind egresses. We should start with a Kafka one, as that is the currently only supported ingress for multi-language applications.
We propose to initially add a more "manual" approach for this, which gives the user full freedom on partitioning and record serialization.
A more specialized egress with built-in routing / protobuf serialization can maybe be added in the future once we collect more knowledge on usage from users.
The plan is as follows:
- Add a KafkaProducerRecord protobuf message with schema:
message KafkaProducerRecord { string topic string key bytes valueBytes }How the value was serialized would be completely user-space; the bytes are written as is to Kafka. Partitioning depends on the partitioner used by the Kafka client, which may be configured with Kafka properties via YAML.
- The Flink sink provider for this egress should expect only messages of KafkaProducerRecord (wrapped within a protobuf Any). It simply transforms the information in the received KafkaProducerRecord to Kafka's own ProducerRecord, and writes that to Kafka.
