Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
We've encountered a number of issues with the current SystemProducer interface with async processors. For example, see SAMZA-1392.
Most of the issues arise from the fact that there's no way for us to notify the user when there is an async error in from the producer. Consider the KafkaSystemProducer. If the KafkaProducer callback returns an exception, the KafkaSystemProducer has to track it and find an appropriate time to handle the exception. Depending on where the exception is handled and whether it is swallowed the OffsetManager will need to be overly conservative to ensure we do not checkpoint offsets until the corresponding send() was successful. Sometimes whole batches of messages will fail, but only a single exception is thrown, and there's no context from which send() the exception originated. This will always be unintuitive to the user unless they are synchronously notified of the exception.
There hasn't been much planning on this yet, but the idea is to have a send() method that takes a callback which is called directly from the producer callback. So the callback would flow something like this:
producer.callback -> user.callback -> asynctask.callback -> (throw error | offsetmanager.update)
Some care needs to be taken to ensure that there are no gaps in the offsets recorded by the OffsetManager. Also, it's not yet clear how to enable users to retry a failed send.
NOTE: depending on the implementation, the changes in SAMZA-1384 may need to be reverted after this feature is implemented to ensure that the latest offsets are checkpointed.