Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-12480

Reuse bootstrap servers in clients when last alive broker in cluster metadata is unavailable

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: clients
    • Labels:
      None

      Description

      https://issues.apache.org/jira/browse/KAFKA-12455 documented how a Java client can temporarily lose connectivity to a 2-broker cluster that is undergoing a roll because the client will repeatedly retry connecting to the last alive broker that it knows about in the cluster metadata even when that broker is unavailable. The client could potentially fallback to its bootstrap brokers in this case and reconnect to the cluster quicker.

      For example, assume a 2-broker cluster has broker IDs 1 and 2 and both appear in the bootstrap servers for a consumer. Assume broker 1 rolls such that the Java consumer receives a new METADATA response and only knows about broker 2 being alive, and then broker 2 rolls before the consumer gets a new METADATA response indicating that broker 1 is also alive. At this point the Java consumer will keep retrying broker 2, and it will not reconnect to the cluster unless/until broker 2 becomes available – or the client itself is restarted so it can use its bootstrap servers again. Another possibility is to fallback to the full bootstrap servers list when the last alive broker becomes unavailable.

      I believe librdkafka-based client may perform this fallback, though I am not certain. We should consider it for Java clients.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              rndgstn Ron Dagostino

              Dates

              • Created:
                Updated:

                Issue deployment