[SOLR-15089] Allow backup/restoration to Amazon's S3 blobstore

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Fix Version/s: 8.10
• Component/s: contrib - S3 Repository
• Assignee: Houston Putman
• Reporter: Jason Gerlowski
• Created: 19/Jan/21 16:03
• Updated: 19/Oct/21 15:32
• Resolved: 24/Aug/21 15:50

Description

Solr's BackupRepository interface provides an abstraction around the physical location/format that backups are stored in. This allows plugin writers to create "repositories" for a variety of storage mediums. It'd be nice if Solr offered more mediums out of the box though, such as some of the "blobstore" offerings provided by various cloud providers.

This ticket proposes that a "BackupRepository" implementation for Amazon's popular 'S3' blobstore, so that Solr users can use it for backups without needing to write their own code.

Amazon offers a s3 Java client with acceptable licensing, and the required code is relatively simple. The biggest challenge in supporting this will likely be procedural - integration testing requires S3 access and S3 access costs money. We can check with INFRA to see if there is any way to get cloud credits for an integration test to run in nightly Jenkins runs on the ASF Jenkins server. Alternatively we can try to stub out the blobstore in some reliable way.

Issue Links

• is related to: SOLR-15599 Upgrade AWS SDK from v1 to v2 for the S3 Repository
• relates to: SOLR-15051 Shared storage -- BlobDirectory (de-duping)
• supercedes: SOLR-9952 S3BackupRepository
• links to: GitHub Pull Request #120
• links to: GitHub Pull Request #2554