[OAK-7382] Cloud datastore without local disk

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: blob, blob-cloud
• Assignee: Amit Jain
• Reporter: Thomas Mueller
• Created: 29/Mar/18 14:40
• Updated: 26/May/21 14:52

Description

Currently, the S3 datastores need local disk to work (not sure about the Azure one).

This should not be needed (not for upload, caching,...).

Also, temporary files for garbage collection should not be needed (instead, use temporary binaries, possibly written to S3 / Azure).

Really everything should fit in a few MB of memory.

For S3, it might be needed to read a few MB of data into memory, and then possibly do a multipart upload:

https://stackoverflow.com/questions/8653146/can-i-stream-a-file-upload-to-s3-without-a-content-length-header