[HDDS-8238] Multipart upload generates a huge OM rocksdb log

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: OM
• Labels: ozone-performance
• Assignee: Unassigned
• Reporter: Tsz-wo Sze
• Created: 22/Mar/23 08:27
• Updated: 06/Nov/24 23:14

Description

When using multipart upload to upload a 1000-part key. OM RocksDb log needs to use 733.15 MB disk space. For a 10,000-part key, it can use 71.5 GB .  The disk space usage for the log is O(n^2) for a n-part key.

MultipartKeyInfo has a list of PartKeyInfo (partKeyInfoList) while each PartKeyInfo has a KeyInfo and the size of each KeyInfo is ~1.5 KB.

//OmClientProtocol.proto
message MultipartKeyInfo {
    required string uploadID = 1;
    ...
    repeated PartKeyInfo partKeyInfoList = 5;
    ...
}

message PartKeyInfo {
    required string partName = 1;
    required uint32 partNumber = 2;
    required KeyInfo partKeyInfo = 3; // the size of a KeyInfo is ~1.5 KB
}

Multipart upload repeatedly puts the same key with a MultipartKeyInfo object for each part.

• When uploading the first part, partKeyInfoList has 1 element.
• When uploading the second part, partKeyInfoList has 2 elements.
• When uploading the third part, partKeyInfoList has 3 elements.
  ...

• When uploading the 1000th part, partKeyInfoList has 1000 elements.

The partKeyInfoList is written to RocksDb log for each part. The number of PartKeyInfo objects written to RocksDb log is

• (1 + 1000)*1000/2 = 500,500 objects

and the size is 1.5 KB * 500,500 = 733.15 MB.

Note that only the RocksDb log size is large. The size of the RocksDb tables are small since the partKeyInfoList will be overwritten again and again for each part.

Issue Links

• is related to: HDDS-11655 Prune redundant KeyInfo fields from PurgeDirectories request
• relates to: HDDS-10611 Optimization of gc pressure on om due to handling of mpu large files
• relates to: HDDS-8128 Deduplicate the ops in RDBBatchOperation
• relates to: HDDS-8427 Optimize multipartUpload memory usage
• relates to: HDDS-1186 Ozone S3 gateway (phase III)