[OAK-4837] Improved caching for DataStore

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: blob
• Labels: performance
• Epic Link: Datastore performance improvements
• Assignee: Amit Jain
• Reporter: Amit Jain
• Created: 22/Sep/16 03:55
• Updated: 08/Oct/19 15:21
• Resolved: 20/Dec/16 03:11

Description

The current CachingDataStore implementation used with S3DataStore has certain problems:

• Lack of stats to show hit rate/miss rates for files being requested for downloads
• Lack of stats for async uploads
• CachingDataStore starts proactively downloading files in the background when a call to getRecord is made.
• Async upload functionality leaks into Backend implementations, LocalCache classes.
• The call to DataStore#getRecord() which makes multiple calls to backends which is problematic for S3 (i.e. when not being served bu cache)
• There is some functionality which is not used with Oak like length cache, sync/async touch etc. which can be removed and code simplified.

Issue Links

• relates to: OAK-3090 Caching BlobStore implementation

Sub-Tasks

• Move S3 classes to oak-blob-cloud module: Closed
• Improve oak-blob-cloud tests: Closed
• Update SegmentS3DataStoreBlobGCTest in oak-it once oak-segment-tar updated: Closed
• Enable benchmarks for DataStore: Closed
• Implement caching for S3DataStore: Closed
• Implement caching for FileDataStore: Closed
• Caching sub-system implementation for DataStore: Closed
• Retry mechanism for failed async uploads: Closed
• Improve normalization of configured path in AbstractSharedCachingDataStore: Closed
• Support upgrade of DataStore cache: Closed