Details
-
Sub-task
-
Status: Closed
-
Major
-
Resolution: Fixed
-
2.7.4
-
None
Description
S3DataStore Asynchronous Upload to S3
The current logic to add a file record to S3DataStore is first add the file in local cache and then upload that file to S3 in a single synchronous step. This feature contemplates to break the current logic with synchronous adding to local cache and asynchronous uploading of the file to S3. Till asynchronous upload completes, all data (inputstream, length and lastModified) for that file record is fetched from local cache.
AWS SDK provides upload progress listeners which provides various callbacks on the status of in-progress upload.
As of now customer reported that write performance of EBS based Datastore is 3x better than S3 DataStore.
With this feature, the objective is to have comparable write performance of S3 DataStore.
Attachments
Issue Links
- is related to
-
OAK-4440 Async upload in S3DataStore can lead to intermittent binary missing exception in cluster
- Resolved