Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
-
None
Description
The embedded consumer in the kafka mirror implementation uses the low-level Log api to create the replica. The append operation may fail and result in a corrupt log file, due to an IO error which is currently caught and ignored.
The proposed fix is to switch to using the high-level producer API to create the replica. Not only would this avoid the above issue, but it would also fit better with the current design of the replication enhancement for kafka (http://linkedin.jira.com/browse/KAFKA-23), since the low-level Log api is not replication-aware. Another advantage is that compression is exposed at the producer API-level. One caveat in this approach would be the following: the async producer drops events when its queue is full. This behavior is unsuitable for the embedded consumer, so we can expose a configuration option in the producer to allow for (queue-level) blocking semantics.