1. Kafka
  2. KAFKA-1025

Producer.send should provide recoverability info on failiure


    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.8.0
    • Fix Version/s:
    • Component/s: None
    • Labels:


      Currently, in 0.8, the Producer.send() method either succeeds, or fails by throwing an Exception.

      There are several exceptions that can be thrown, including:


      These are all sub-classes of RuntimeException.

      Under the covers, the producer will retry sending messages up to a maximum number of times (according to the message.send.max.retries property). Internally, the producer may decide which sorts of failures are recoverable, and will retry those. Alternatively (via an upcoming change, see KAFKA-998), it may decide to not retry at all, if the error is not recoverable.

      The problem is, if FailedToSendException is returned, the caller to Producer.send doesn't have a way to decide if a send failed due to an unrecoverable error, or failed after exhausting a maximum number of retries.

      A caller may want to decide to retry more times, perhaps after waiting a while. But it should know first whether it's even likely that the failure is retryable.

      An example of this might be a if the message size is too large (represented internally as a MessageSizeTooLargeException). In this case, it is not recoverable, but it is still wrapped as a FailedToSendException, and should not be retried.

      So the suggestion is to make clear in the api javadoc (or scaladoc) for Producer.send, the set of exception types that can be thrown (so that we don't have to search through source code to find them). And add exception types, or perhaps fields within FailedToSendException, so that it's possible to reason about whether retrying might make sense.

      Currently, in addition, I've found that Producer.send can throw a QueueFullException in async mode (this should be a retryable exception, after time has elapsed, etc.), and also a ClassCastException, if there's a misconfiguration between the configured Encoder and the message data type. I suspect there are other RuntimeExceptions that can also be thrown (e.g. NullPointerException if the message/topic are null).


        Jason Rosenberg created issue -
        Neha Narkhede made changes -
        Field Original Value New Value
        Labels newbie
        Dima Pekar made changes -
        Attachment KAFKA-1025.patch [ 12622403 ]
        Dima Pekar made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Joe Stein made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Fix Version/s 0.8.2 [ 12326167 ]
        Joe Stein made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Won't Fix [ 2 ]
        Tony Stevenson made changes -
        Workflow no-reopen-closed, patch-avail [ 12812742 ] Apache Kafka Workflow [ 13051465 ]
        Tony Stevenson made changes -
        Workflow Apache Kafka Workflow [ 13051465 ] no-reopen-closed, patch-avail [ 13054648 ]


          • Assignee:
            Jason Rosenberg
          • Votes:
            1 Vote for this issue
            6 Start watching this issue


            • Created: