[HDDS-7814] Implement remote S3 secret storage

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Affects Version/s: 1.4.0
• Fix Version/s: 1.4.0
• Component/s: S3
• Labels: pull-request-available
• Assignee: Mikhail Pochatkin
• Reporter: Mikhail Pochatkin
• Created: 20/Jan/23 12:07
• Updated: 07/Sep/23 08:58
• Resolved: 08/May/23 12:41

Description

The S3 secrets are currently stored in the RocksDB of the Ozone manager nodes. With this approach, it is not possible to separate the storage of secrets from nodes with an ozone manager. This is a limitation in some environments, for various reasons, such as security issues, so it is proposed to add the ability to store secrets separately from the ozone managers. One of the options for storing secrets would be to use a third-party solution, an example of HashiСorp Vault . Therefore, it is proposed to add the implementation of the storage of S3 secrets based on a remote http server. It is proposed to configure the type of storage using a special property in the ozone site. Leave the current RocksDB as the default implementation to maintain backwards compatibility.

Issue Links

• is a child of: HDDS-8132 Secure S3 keys management
• is a parent of: HDDS-7815 Extract layer of S3 secret manipulation
• relates to: HDDS-8050 Create HTTP endpoint to access S3 secret via Kerberos auth
• relates to: HDDS-8640 Support multiple acceptance test scripts with custom config
• relates to: HDDS-9250 Test failure in VaultS3SecretStoreTest#testAuthFail
• links to: GitHub Pull Request #4389