Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
While verifying the 10.1 release with a test job, I started getting a CORRUPT_MESSAGE exception from Kafka while flushing the changelog.
Turns out, the job was writing an empty string as a Key in the store. After serialization, the string translated to an empty byte[], which is not allowed for log compacted topics.
The error message isn't intuitive for users. Ideally we'd catch the issue in the KV store, but the key hasn't been serialized at that point. The next best place to catch it is right after serialization, where we can measure the key size and throw if sending to a log compacted topic.
We could add this check in the current KafkaSystemProducer, but there is discussion about moving to the high-level kafka system producer which, I believe, performs serialization. We should coordinate with the kafka team to see if they have a more useful error message in the high-level producer.