Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3177

Kafka consumer can hang when position() is called on a non-existing partition.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.9.0.0
    • Fix Version/s: 2.0.0
    • Component/s: clients
    • Labels:
      None

      Description

      This can be easily reproduced as following:

      {
          ...
          consumer.assign(SomeNonExsitingTopicParition);
          consumer.position();
          ...
      }
      

      It seems when position is called we will try to do the following:
      1. Fetch committed offsets.
      2. If there is no committed offsets, try to reset offset using reset strategy. in sendListOffsetRequest(), if the consumer does not know the TopicPartition, it will refresh its metadata and retry. In this case, because the partition does not exist, we fall in to the infinite loop of refreshing topic metadata.

      Another orthogonal issue is that if the topic in the above code piece does not exist, position() call will actually create the topic due to the fact that currently topic metadata request could automatically create the topic. This is a known separate issue.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hachikuji Jason Gustafson
                Reporter:
                becket_qin Jiangjie Qin
                Reviewer:
                Jason Gustafson
              • Votes:
                4 Vote for this issue
                Watchers:
                18 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: