[HDDS-4] Implement security for Hadoop Distributed Storage Layer

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.4.0
• Component/s: Security
• Sprint: HDDS BadLands
• Assignee: Xiaoyu Yao
• Reporter: Anu Engineer
• Created: 22/Feb/18 23:54
• Updated: 14/Oct/22 21:09
• Resolved: 22/May/19 16:00

Description

In HDFS-7240, we have created a scalable block layer that facilitates separation of namespace and block layer. Hadoop Distributed Storage Layer (HDSL) allows us to scale HDFS(HDFS-10419) and as well as create ozone (HDFS-13074).

This JIRA is an umbrella JIRA that tracks the security-related work items for Hadoop Distributed Storage Layer.

Issue Links

• relates to: HDFS-7240 Scaling HDFS
• relates to: HDDS-7331 Ozone PKI improvements

Sub-Tasks

• Enable OzoneManager kerberos auth: Resolved
• Enable SCM kerberos auth: Resolved
• Enable kerberos auth for Ozone client in hadoop rpc: Resolved
• Fix config names for secure ksm and scm: Resolved
• Add kdc docker image for secure ozone cluster: Resolved
• Adding Ozone Manager Audit Log: Resolved
• Fix secure docker and configs: Resolved
• Move OzoneSecure docker-compose after HDDS-447: Resolved
• Resolve bouncy castle dependency for hadoop-hdds-common: Resolved
• SCM CA: generate public/private key pair for SCM/OM/DNs: Resolved
• Create a Self-Signed Certificate: Resolved
• SelfSignedCertificate#generateCertificate should sign the certificate the configured security provider: Resolved
• Adding ASF license header to kadm5.acl: Resolved
• Fix the Dependency convergence issue on HDDS-4: Resolved
• Fix HDDS-4 branch after HDDS-490 and HADOOP-15832: Resolved
• SCM CA: Add new security protocol for SCM to expose security related functions: Resolved
• SCM CA: generate CSR for SCM CA clients: Resolved
• Add asf license to TestCertificateSignRequest: Resolved
• SCM security protocol server is not starting: Resolved
• Fix ozone-secure.robot test: Resolved
• Add an interface for CA and Clients for Certificate operations: Resolved
• Add TokenIdentifier Ozone for delegation token and block token: Resolved
• SCM CA: Update DelegationKey to retrieve private/public key: Resolved
• Bootstrap OM/SCM with private/public key pair: Resolved
• Add OzoneManager Delegation Token support: Resolved
• Add GRPC protocol interceptors for Ozone Block Token: Resolved
• Fix TestSecureOzoneContainer NPE after HDDS-837: Resolved
• Bootstrap genesis SCM(CA) with self-signed certificate.: Resolved
• Block token: Add secret token manager: Resolved
• Fix merge issue that causes NPE OzoneManager#httpServer: Resolved
• GRPC: Support secure gRPC endpoint with mTLS: Resolved
• Document ozone.max.key.len usage: Resolved
• Remove ozone.max.key.len property: Resolved
• Block token: Client api changes for block token: Resolved
• Create an S3 Auth Table: Resolved
• SCM CA: SCM CA server signs certificate for approved CSR: Resolved
• SCM CA: Add CA to SCM.: Resolved
• Fix generics warnings in delegation token: Resolved
• Add Client APIs for using S3 Auth interface: Resolved
• Fix failure in TestOzoneShell due to null check in SecurityConfig: Resolved
• Fix test failure in TestOmMetrics: Resolved
• Fix TestOzoneConfigrationFields: Resolved
• Fix classnotfound error for bouncy castle classes in OM,SCM init: Resolved
• Fix failures in TestOzoneConfigurationFields: Resolved
• Unblock certain SCM client APIs from SCM#checkAdminAccess: Resolved
• Ratis: Support secure gRPC endpoint with mTLS for Ratis: Resolved
• Add cli command option for getS3Secret: Resolved
• Fix TestOzoneManagerRatisServer.testIsReadOnlyCapturesAllCmdTypeEnums: Resolved
• Add block token validation in HddsDispatcher/XceiverServer: Resolved
• Manage ozone security tokens with ozone shell cli: Resolved
• Adding getOMCertificate in SCMSecurityProtocol: Resolved
• GRPC: DN changes to use cert issued by SCM for GRPC mTLS: Resolved
• Add Default CertificateClient implementation: Resolved
• Implement OM init in secure cluster: Resolved
• Bootstrap DN with private/public key pair: Resolved
• OzoneManager fails to connect with secure SCM: Resolved
• Fix findbugs issues in DefaultCertificateClient#handleCase: Resolved
• OzoneManager need to login during init when security is enabled.: Resolved
• SCM CA: Write Certificate information to SCM Metadata: Resolved
• Add API to get OM certificate from SCM CA: Resolved
• Support Service Level Authorization for Ozone: Resolved
• Allow persisting X509CertImpl to SCM certificate table: Resolved
• DelegationToken: Add certificate serial id to Ozone Delegation Token Identifier: Resolved
• Fix jdk 11 issue for ozonesecure base image and docker-compose: Resolved
• Fix ClassNotFound issue with javax.xml.bind.DatatypeConverter used by DefaultProfile: Resolved
• SCM CA: OM sends CSR and uses certificate issued by SCM: Resolved
• Support getDelegationToken API for OzoneFileSystem: Resolved
• OzoneManager NPE reading private key file.: Resolved
• Change name of ozoneManager service in docker compose files to om: Resolved
• SCM CA: DN sends CSR and uses certificate issued by SCM: Resolved
• BaseHttpServer NPE is HTTP policy is HTTPS_ONLY: Resolved
• Add robot test for OM Block Token: Resolved
• Fix incorrect Ozone ClientProtocol KerberosInfo annotation: Resolved
• OM delegation expiration time should use Time.now instead of Time.monotonicNow: Resolved
• Fix checkstyle issue from Nightly run: Resolved
• Enable token based authentication for S3 api: Resolved
• Fix TestDefaultCertificateClient#testSignDataStream: Resolved
• Fix failure in TestOzoneManagerHttpServer & TestStorageContainerManagerHttpServer: Resolved
• Add robot test for OM Delegation Token: Resolved
• Add ozone delegation token utility subcmd for Ozone CLI: Resolved
• Fix checkstyle issue from Nightly run: Resolved
• DN get OM certificate from SCM CA for block token validation: Resolved
• Refactor om token db as column family.: Resolved
• Enable service policy authorization for HDDS group: Resolved
• OM get the certificate from SCM CA for token validation: Resolved
• Change hadoop-runner and apache/hadoop base image to use Java8: Resolved
• Update MiniOzoneCluster to work with security protocol from SCM: Resolved
• Support TokenIssuer interface for running jobs with OzoneFileSystem: Resolved
• Set OmKeyArgs#refreshPipeline flag properly to avoid reading from stale pipeline: Resolved
• Refactor ozone acceptance test to allow run in secure mode: Resolved
• Ozone serialization codec for Ozone S3 secret table: Resolved
• KeyOutputStream#write throws ArrayIndexOutOfBoundsException when running RandomWrite MR examples: Resolved
• Fix MalformedTracerStateStringException on DN logs: Resolved
• Spark job fails to create ozone rpc client: Resolved
• NPE if secure ozone if KMS uri is not defined.: Resolved
• ozone spark job failing with class not found error for hadoop 2: Resolved
• Inconsistent naming convention with Ozone Kerberos configuration: Resolved
• Update ratis dependency to 0.3.0: Resolved
• Add retry to kinit command in smoketests: Resolved