[OAK-8551] Minimize network calls in cloud data stores (performance optimization)

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.16.0, 1.18.0, 1.10.4
• Fix Version/s: 1.20.0
• Component/s: blob-cloud, blob-cloud-azure
• Assignee: Matt Ryan
• Reporter: Matt Ryan
• Created: 16/Aug/19 20:46
• Updated: 05/Nov/19 08:33
• Resolved: 05/Nov/19 01:49

Description

Oak cloud data stores (e.g. AzureDataStore, S3DataStore) are by definition more susceptible to performance degradation due to network issues.  While we can't do much about the performance of uploading or downloading a blob, there are other places within the implementations where we are making network calls to the storage service which might be avoidable or minimized.

One example is the exists() call to check whether a blob with a particular identifier exists in the blob storage.  In some places exists() is being called where instead we could simply attempt the network access and handle failures elegantly, avoiding making an extra network call.  In other places perhaps a cache could be used to minimize round trips.

Another example is the higher-level getReference() call in DataStoreBlobStore.  This asks the implementation for a DataRecord and then gets the reference from that, but in truth the data store backend can already obtain a reference for an identifier on its own.  Asking for the DataRecord however requires a network request to get the blob metadata for the record.

Issue Links

• relates to: OAK-7998 [DirectBinaryAccess] Verify that binary exists in cloud before creating signed download URI

Sub-Tasks

• Minimize network calls required when creating a direct download URI: Closed
• Minimize network calls required when calling getRecord(): Closed
• Minimize network calls required when completing direct upload: Closed
• Add instrumentation to track cloud data store usage: Closed
• Add instrumentation to S3DataStore: Resolved
• Introduce API to check whether blob inlined in Id: Closed