Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7739 Kafka Tiered Storage
  3. KAFKA-15181

Race condition on partition assigned to TopicBasedRemoteLogMetadataManager

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.6.0
    • core

    Description

      TopicBasedRemoteLogMetadataManager (TBRLMM) uses a cache to be prepared whever partitions are assigned.

      When partitions are assigned to the TBRLMM instance, a consumer is started to keep the cache up to date.

      If the cache hasn't finalized to build, TBRLMM fails to return remote metadata about partitions that are store on the backing topic. TBRLMM may not recover from this failing state.

      A proposal to fix this issue would be wait after a partition is assigned for the consumer to catch up. A similar logic is used at the moment when TBRLMM writes to the topic, and uses send callback to wait for consumer to catch up. This logic can be reused whever a partition is assigned, so when TBRLMM is marked as initialized, cache is ready to serve requests.

      Reference: https://github.com/aiven/kafka/issues/33

      Attachments

        Activity

          People

            abhijeetkumar Abhijeet Kumar
            jeqo Jorge Esteban Quilcate Otoya
            Votes:
            2 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: