[KAFKA-7739] Kafka Tiered Storage

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.6.0
• Component/s: core
• Labels: KIP-405
• Assignee: Satish Duggana
• Reporter: Harsha
• Created: 14/Dec/18 19:51
• Updated: 13/Dec/23 06:43
• Resolved: 14/Sep/23 14:08

Description

KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage

Next version of tiered storage is tracked at https://issues.apache.org/jira/browse/KAFKA-15420

Issue Links

• duplicates: KAFKA-7374 Tiered Storage
• is a child of: KAFKA-15386 Update log-start-offset in RLM task for a leader partition for the earliest leader epoch of the current leader's leader epoch lineage available in the remote storage.
• is a parent of: KAFKA-13097 Handle the requests gracefully to publish the events in TopicBasedRemoteLogMetadataManager when it is not yet initialized.
• is a parent of: KAFKA-15290 Add support to onboard existing topics to tiered storage
• is a parent of: KAFKA-15189 Do not initialize RemoteStorage related metrics when disabled at cluster
• is a parent of: KAFKA-15135 RLM listener configurations passed but ignored by RLMM
• is a parent of: KAFKA-14993 Improve TransactionIndex instance handling while copying to and fetching from RSM.
• is a parent of: KAFKA-14038 Optimize calculation of size for log in remote tier
• is a parent of: KAFKA-15130 Delete remote segments when delete a topic
• is a parent of: KAFKA-15131 Improve RemoteStorageManager exception handling documentation
• is a parent of: KAFKA-15194 Rename local tiered storage segment with offset as prefix for easy navigation
• is related to: KAFKA-15414 remote logs get deleted after partition reassignment
• is related to: KAFKA-15274 support moving files to be deleted to other directories
• is required by: KAFKA-15132 Implement disable & re-enablement for Tiered Storage
• mentioned in: Page Loading...
• mentioned in: Page Loading...
• mentioned in: Page Loading...

Sub-Tasks

• SPI - RemoteStorageManager and RemoteLogMetadataManager interfaces and related classes.: Resolved
• Local storage implementations for RSM which can be used in tests: Resolved
• RemoteLogManager - copying eligible log segments to remote storage implementation: Resolved
• Define the SPI for Tiered Storage framework: Resolved
• Topic-based implementation for the RemoteLogMetadataManager: Resolved
• Integration Test framework for Tiered Storage: Resolved
• Implementation of Tiered Storage SPI to integrate with S3: Resolved
• RemoteStorageManager implementation for HDFS storage.: Resolved
• Remote consumer fetch implementation by adding respective purgatory: Resolved
• Supporting transactions in tiered storage: Resolved
• Inmemory implementation of RSM and RLMM.: Resolved
• Serdes for all message types in internal topic which is used in default implementation for RLMM.: Resolved
• Implementation of Tiered Storage Integration with Azure Storage (ADLS + Blob Storage): Resolved
• Add a file based cache for consumed remote log metadata for each partition to avoid consuming again incase of broker restarts.: Resolved
• Add tier storage configs.: Resolved
• Add cluster or broker level config for topic level tiered storage confgs.: Resolved
• Make tiered storage related schemas adopt flexible versions feature.: Resolved
• Change RLMM add/updateRemoteLogSegmentMetadata and putRemotePartitionDeleteMetadata APIS asynchronous.: Resolved
• Handle the requests gracefully to publish the events in TopicBasedRemoteLogMetadataManager when it is not yet initialized.: Resolved
• Follower fetch protocol enhancements for tiered storage.: Resolved
• Refactor ClassLoaderAwareRemoteStorageManager.scala to ClassLoaderAwareRemoteStorageManager.java and move it to storage module.: Resolved
• Fix inconsistency in config name between code and KIP-405: Resolved
• Improve the RemoteIndexCacheTest: Resolved
• TierStateMachine interface for building remote aux log: Resolved
• Introduce InMemoryLeaderEpochCheckpoint to allow quick write/read: Resolved
• RemoteLogManager - deleting expired/size breached log segments to remote storage implementation: Resolved
• RemoteLogManager - allow consumer fetch records from remote storage implementation: Resolved
• Add metrics for tiered storage: Resolved
• initialize unifiedLog with remoteStorageSystemEnable correctly: Resolved
• segment copy to remote storage won't work in KRaft mode: Resolved
• passing listener name config into TopicBasedRemoteLogMetadataManagerConfig: Resolved
• Passing "remote.log.metadata.*" configs into RLMM: Resolved
• Remove lock contention in RemoteIndexCache: Resolved
• Print startup time for RemoteIndexCache: Resolved
• Tiered Storage Test Harness Framework: Resolved
• Handle overlapping remote log segments in RemoteLogMetadata cache: Resolved
• Add missing tests for remote storage metrics: Resolved
• Race condition on partition assigned to TopicBasedRemoteLogMetadataManager: Resolved
• Rename Remote Storage metrics to remove ambiguity: Resolved
• RLM Task should wait until RLMM is initialized before copying segments to remote: Resolved
• ReplicaFetcher thread should not block if RLMM is not initialized: Resolved
• Cluster-wide disablement of Tiered Storage: Resolved
• Add support to onboard existing topics to tiered storage: Resolved
• Update metrics doc to add tiered storage metrics: Resolved
• Make remote storage related configs as public (i.e. non-internal): Resolved
• Add config validation when remote storage is enabled on a topic: Resolved
• Disable DeleteRecords on Tiered Storage topics: Resolved
• Make default `remote.log.metadata.manager.class.name` as topic based RLMM: Resolved
• Update log-start-offset after leader election for topics enabled with remote storage: Resolved
• Ensure consistency while deleting the remote log segments: Resolved
• Enable OffloadAndConsumeFromLeader test: Resolved
• Add functional integration tests with tiered storage: Resolved
• Integration tests in TS test harness detect resource leaks: Resolved
• add document to introduce tiered storage feature and the usage: Resolved