[KAFKA-15682] Ensure internal remote log metadata topic does not expire its segments before deleting user-topic segments

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Kamal Chandraprakash
• Created: 25/Oct/23 12:49
• Updated: 13/Jun/24 00:44

Description

One of the implementation of RemoteLogMetadataManager is TopicBasedRemoteLogMetadataManager which uses an internal Kafka topic __remote_log_metadata to store the metadata about the remote log segments. Unlike other internal topics which are compaction enabled, this topic is not enabled with compaction and retention is set to unlimited.

Keeping this internal topic retention to unlimited is not practical in real world use-case where the topic local disk usage footprint grow huge over a period of time.

It is assumed that the user will set the retention to a reasonable time such that it is the max of all the user-created topics (max + X). We can't just rely on the assumption and need an assertion to ensure that the internal __remote_log_metadata segments are not eligible for deletion before the expiry of all the relevant user-topic uploaded remote-log-segments , otherwise there will be dangling remote-log-segments which won't be cleared once all the brokers are restarted post the internal topic retention cleanup.

See the discussion thread: https://github.com/apache/kafka/pull/14576#discussion_r1368576126

Issue Links

• is a child of: KAFKA-16947 Kafka Tiered Storage V2