Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
When using multipart upload to upload a 1000-part key. OM RocksDb log needs to use 733.15 MB disk space. For a 10,000-part key, it can use 71.5 GB . The disk space usage for the log is O(n^2) for a n-part key.
MultipartKeyInfo has a list of PartKeyInfo (partKeyInfoList) while each PartKeyInfo has a KeyInfo and the size of each KeyInfo is ~1.5 KB.
//OmClientProtocol.proto message MultipartKeyInfo { required string uploadID = 1; ... repeated PartKeyInfo partKeyInfoList = 5; ... } message PartKeyInfo { required string partName = 1; required uint32 partNumber = 2; required KeyInfo partKeyInfo = 3; // the size of a KeyInfo is ~1.5 KB }
Multipart upload repeatedly puts the same key with a MultipartKeyInfo object for each part.
- When uploading the first part, partKeyInfoList has 1 element.
- When uploading the second part, partKeyInfoList has 2 elements.
- When uploading the third part, partKeyInfoList has 3 elements.
...
- When uploading the 1000th part, partKeyInfoList has 1000 elements.
The partKeyInfoList is written to RocksDb log for each part. The number of PartKeyInfo objects written to RocksDb log is
- (1 + 1000)*1000/2 = 500,500 objects
and the size is 1.5 KB * 500,500 = 733.15 MB.
Note that only the RocksDb log size is large. The size of the RocksDb tables are small since the partKeyInfoList will be overwritten again and again for each part.