Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-788

Periodic refresh of topic metadata on the producer doesn't include all topics

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Auto Closed
    • Affects Version/s: 0.8.0
    • Fix Version/s: None
    • Component/s: producer
    • Labels:

      Description

      We added a patch to the producer to refresh the metadata for all topics periodically. However, the producer only does this for the topics in the last batch. But some topics sent by the producer could be low throughput and might not be present in every batch. If we bounce the cluster or if brokers fail and leaders change, the metadata for those low throughput topic is not refreshed by this periodic topic metadata request. The next produce request for those topics have to fail and then a separate metadata request needs to be reissued to handle the produce request. This is especially a problem for the migration tool. So even if the producer had a chance to refresh the metadata when the leader changed, it throws LeaderNotAvailableExceptions much later when it sends a request for that topic.

      I propose we just fetch data for all topics sent by the producer in the periodic refresh of topic metadata

        Activity

        Hide
        omkreddy Manikumar added a comment -

        Closing inactive issue. The old producer is no longer supported.

        Show
        omkreddy Manikumar added a comment - Closing inactive issue. The old producer is no longer supported.

          People

          • Assignee:
            nehanarkhede Neha Narkhede
            Reporter:
            nehanarkhede Neha Narkhede
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 24h
              24h
              Remaining:
              Remaining Estimate - 24h
              24h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development