Kafka
  1. Kafka
  2. KAFKA-1124

Sending to a new topic (with auto.create.topics.enable) returns ERROR

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0, 0.8.1
    • Fix Version/s: 0.8.2.0
    • Component/s: None
    • Labels:

      Description

      I had thought this was reported issue, but can't seem to find a previous report for it.

      If auto.create.topics.enable is true, a producer still gets an ERROR logged on the first attempt to send a message to a new topic, e.g.:

      2013-11-06 03:00:08,638 ERROR [Thread-1] async.DefaultEventHandler - Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: mynewtopic
      2013-11-06 03:00:08,638 INFO [Thread-1] async.DefaultEventHandler - Back off for 100 ms before retrying send. Remaining retries = 3

      This usually clears itself up immediately on retry (after 100 ms), as handled by the the kafka.producer.async.DefaultEventHandler (with retries enabled).

      However, this is logged to the client as an ERROR, and looks scary, when in fact it should have been a normal operation (since we have auto.create.topics.enable=true).

      There should be a better interaction here between the producer client and the server.

      Perhaps the server can create the topic in flight before returning the metadata request.

      Or, if it needs to be asynchronous, it could return a code which indicates something like: "The topic doesn't exist yet, it is being created, try again shortly".....and have the client automatically retry (even if retries not enabled, since it's not an ERROR condition, really).

      The ERROR log level is a problem since apps often have alert systems set up to notify when any ERROR happens, etc.....

        Activity

        Jason Rosenberg created issue -
        Neha Narkhede made changes -
        Field Original Value New Value
        Affects Version/s 0.8.1 [ 12322960 ]
        Neha Narkhede made changes -
        Labels usability
        Manikumar Reddy made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 0.8.2 [ 12326167 ]
        Resolution Fixed [ 1 ]
        Tony Stevenson made changes -
        Workflow no-reopen-closed, patch-avail [ 12824951 ] Apache Kafka Workflow [ 13052019 ]
        Tony Stevenson made changes -
        Workflow Apache Kafka Workflow [ 13052019 ] no-reopen-closed, patch-avail [ 13054193 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Jason Rosenberg
          • Votes:
            2 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development