[KAFKA-15107] Additional custom metadata for remote log segment

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 3.6.0
• Component/s: core
• Labels: tiered-storage
• Assignee: Ivan Yurchenko
• Reporter: Ivan Yurchenko
• Created: 19/Jun/23 11:26
• Updated: 05/Aug/23 09:54
• Resolved: 04/Aug/23 13:33

Description

Based on the KIP-917: Additional custom metadata for remote log segment, the following needs to be implemented:

1. RemoteLogSegmentMetadata.CustomMetadata.
2. RemoteStorageManager.copyLogSegmentData needs to be updated to the new return type (+ javadoc).
3. RemoteLogSegmentMetadata.customMetadata and RemoteLogSegmentMetadata.createWithCustomMetadata methods. Same in RemoteLogSegmentMetadataSnapshot.
4. RemoteLogSegmentMetadataRecord and RemoteLogSegmentMetadataSnapshotRecord definitions need to be updated.
5. Custom metadata should be persisted by RemoteLogManager if provided.
6. The new config remote.log.metadata.custom.metadata.max.size needs to be introduced.
7. The custom metadata size limit must be applied according to the KIP.

Issue Links

• links to: GitHub Pull Request #13984
• mentioned in: Page Loading...