Uploaded image for project: 'Hadoop Distributed Data Store'
  1. Hadoop Distributed Data Store
  2. HDDS-4

Implement security for Hadoop Distributed Storage Layer

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4.0
    • Component/s: Security
    • Labels:
      None
    • Sprint:
      HDDS BadLands

      Description

      In HDFS-7240, we have created a scalable block layer that facilitates separation of namespace and block layer. Hadoop Distributed Storage Layer (HDSL) allows us to scale HDFS(HDFS-10419) and as well as create ozone (HDFS-13074).

      This JIRA is an umbrella JIRA that tracks the security-related work items for Hadoop Distributed Storage Layer.

        Attachments

        1. HadoopStorageLayerSecurity.pdf
          445 kB
          Anu Engineer

          Issue Links

          1.
          Enable OzoneManager kerberos auth Sub-task Resolved Ajay Kumar  
          2.
          Enable SCM kerberos auth Sub-task Resolved Ajay Kumar  
          3.
          Enable kerberos auth for Ozone client in hadoop rpc Sub-task Resolved Ajay Kumar  
          4.
          Fix config names for secure ksm and scm Sub-task Resolved Ajay Kumar  
          5.
          Add kdc docker image for secure ozone cluster Sub-task Resolved Ajay Kumar  
          6.
          Adding Ozone Manager Audit Log Sub-task Resolved Dinesh Chitlangia  
          7.
          Fix secure docker and configs Sub-task Resolved Xiaoyu Yao  
          8.
          Move OzoneSecure docker-compose after HDDS-447 Sub-task Resolved Xiaoyu Yao  
          9.
          Resolve bouncy castle dependency for hadoop-hdds-common Sub-task Resolved Ajay Kumar  
          10.
          SCM CA: generate public/private key pair for SCM/OM/DNs Sub-task Resolved Ajay Kumar  
          11.
          Create a Self-Signed Certificate Sub-task Resolved Anu Engineer  
          12.
          SelfSignedCertificate#generateCertificate should sign the certificate the configured security provider Sub-task Resolved Xiaoyu Yao  
          13.
          Adding ASF license header to kadm5.acl Sub-task Resolved Ajay Kumar  
          14.
          Fix the Dependency convergence issue on HDDS-4 Sub-task Resolved Xiaoyu Yao  
          15.
          Fix HDDS-4 branch after HDDS-490 and HADOOP-15832 Sub-task Resolved Xiaoyu Yao  
          16.
          SCM CA: Add new security protocol for SCM to expose security related functions Sub-task Resolved Ajay Kumar  
          17.
          SCM CA: generate CSR for SCM CA clients Sub-task Resolved Xiaoyu Yao  
          18.
          Add asf license to TestCertificateSignRequest Sub-task Resolved Ajay Kumar  
          19.
          SCM security protocol server is not starting Sub-task Resolved Ajay Kumar  
          20.
          Fix ozone-secure.robot test Sub-task Resolved Ajay Kumar  
          21.
          Add an interface for CA and Clients for Certificate operations Sub-task Resolved Anu Engineer  
          22.
          Add TokenIdentifier Ozone for delegation token and block token Sub-task Resolved Ajay Kumar  
          23.
          SCM CA: Update DelegationKey to retrieve private/public key Sub-task Resolved Ajay Kumar  
          24.
          Bootstrap OM/SCM with private/public key pair Sub-task Resolved Ajay Kumar  
          25.
          Add OzoneManager Delegation Token support Sub-task Resolved Ajay Kumar  
          26.
          Add GRPC protocol interceptors for Ozone Block Token Sub-task Resolved Xiaoyu Yao  
          27.
          Fix TestSecureOzoneContainer NPE after HDDS-837 Sub-task Resolved Xiaoyu Yao  
          28.
          Bootstrap genesis SCM(CA) with self-signed certificate. Sub-task Resolved Anu Engineer  
          29.
          Block token: Add secret token manager Sub-task Resolved Ajay Kumar  
          30.
          Fix merge issue that causes NPE OzoneManager#httpServer Sub-task Resolved Xiaoyu Yao  
          31.
          GRPC: Support secure gRPC endpoint with mTLS Sub-task Resolved Xiaoyu Yao  
          32.
          Document ozone.max.key.len usage Sub-task Resolved Ajay Kumar  
          33.
          Remove ozone.max.key.len property Sub-task Resolved Ajay Kumar  
          34.
          Block token: Client api changes for block token Sub-task Resolved Ajay Kumar  
          35.
          Create an S3 Auth Table Sub-task Resolved Dinesh Chitlangia  
          36.
          SCM CA: SCM CA server signs certificate for approved CSR Sub-task Resolved Anu Engineer  
          37.
          SCM CA: Add CA to SCM. Sub-task Resolved Anu Engineer  
          38.
          Fix generics warnings in delegation token Sub-task Resolved Ajay Kumar  
          39.
          Add Client APIs for using S3 Auth interface Sub-task Resolved Dinesh Chitlangia  
          40.
          Fix failure in TestOzoneShell due to null check in SecurityConfig Sub-task Resolved Ajay Kumar  
          41.
          Fix test failure in TestOmMetrics Sub-task Resolved Ajay Kumar  
          42.
          Fix TestOzoneConfigrationFields Sub-task Resolved Anu Engineer  
          43.
          Fix classnotfound error for bouncy castle classes in OM,SCM init Sub-task Resolved Ajay Kumar  
          44.
          Fix failures in TestOzoneConfigurationFields Sub-task Resolved Ajay Kumar  
          45.
          Unblock certain SCM client APIs from SCM#checkAdminAccess Sub-task Resolved Xiaoyu Yao  
          46.
          Ratis: Support secure gRPC endpoint with mTLS for Ratis Sub-task Resolved Xiaoyu Yao  
          47.
          Add cli command option for getS3Secret Sub-task Resolved Dinesh Chitlangia  
          48.
          Fix TestOzoneManagerRatisServer.testIsReadOnlyCapturesAllCmdTypeEnums Sub-task Resolved Xiaoyu Yao  
          49.
          Add block token validation in HddsDispatcher/XceiverServer Sub-task Resolved Ajay Kumar  
          50.
          Manage ozone security tokens with ozone shell cli Sub-task Resolved Ajay Kumar  
          51.
          Adding getOMCertificate in SCMSecurityProtocol Sub-task Resolved Ajay Kumar  
          52.
          GRPC: DN changes to use cert issued by SCM for GRPC mTLS Sub-task Resolved Xiaoyu Yao  
          53.
          Add Default CertificateClient implementation Sub-task Resolved Ajay Kumar  
          54.
          Implement OM init in secure cluster Sub-task Resolved Ajay Kumar  
          55.
          Bootstrap DN with private/public key pair Sub-task Resolved Ajay Kumar  
          56.
          OzoneManager fails to connect with secure SCM Sub-task Resolved Ajay Kumar  
          57.
          Fix findbugs issues in DefaultCertificateClient#handleCase Sub-task Resolved Ajay Kumar  
          58.
          OzoneManager need to login during init when security is enabled. Sub-task Resolved Xiaoyu Yao  
          59.
          SCM CA: Write Certificate information to SCM Metadata Sub-task Resolved Anu Engineer  
          60.
          Add API to get OM certificate from SCM CA Sub-task Resolved Ajay Kumar  
          61.
          Support Service Level Authorization for Ozone Sub-task Resolved Xiaoyu Yao  
          62.
          Allow persisting X509CertImpl to SCM certificate table Sub-task Resolved Xiaoyu Yao  
          63.
          DelegationToken: Add certificate serial id to Ozone Delegation Token Identifier Sub-task Resolved Ajay Kumar  
          64.
          Fix jdk 11 issue for ozonesecure base image and docker-compose Sub-task Resolved Xiaoyu Yao  
          65.
          Fix ClassNotFound issue with javax.xml.bind.DatatypeConverter used by DefaultProfile Sub-task Resolved Xiaoyu Yao

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          66.
          SCM CA: OM sends CSR and uses certificate issued by SCM Sub-task Resolved Ajay Kumar

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 10m
          67.
          Support getDelegationToken API for OzoneFileSystem Sub-task Resolved Xiaoyu Yao

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h
          68.
          OzoneManager NPE reading private key file. Sub-task Resolved Xiaoyu Yao  
          69.
          Change name of ozoneManager service in docker compose files to om Sub-task Resolved Ajay Kumar

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 50m
          70.
          SCM CA: DN sends CSR and uses certificate issued by SCM Sub-task Resolved Ajay Kumar

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 5h 20m
          71.
          BaseHttpServer NPE is HTTP policy is HTTPS_ONLY Sub-task Resolved Xiaoyu Yao

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          72.
          Add robot test for OM Block Token Sub-task Resolved Ajay Kumar

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1.5h
          73.
          Fix incorrect Ozone ClientProtocol KerberosInfo annotation Sub-task Resolved Xiaoyu Yao

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          74.
          OM delegation expiration time should use Time.now instead of Time.monotonicNow Sub-task Resolved Xiaoyu Yao

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          75.
          Fix checkstyle issue from Nightly run Sub-task Resolved Xiaoyu Yao

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          76.
          Enable token based authentication for S3 api Sub-task Resolved Ajay Kumar

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 29.5h
          77.
          Fix TestDefaultCertificateClient#testSignDataStream Sub-task Resolved Xiaoyu Yao

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          78.
          Fix failure in TestOzoneManagerHttpServer & TestStorageContainerManagerHttpServer Sub-task Resolved Ajay Kumar

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1.5h
          79.
          Add robot test for OM Delegation Token Sub-task Resolved Ajay Kumar  
          80.
          Add ozone delegation token utility subcmd for Ozone CLI Sub-task Resolved Xiaoyu Yao

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 40m
          81.
          Fix checkstyle issue from Nightly run Sub-task Resolved Xiaoyu Yao

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          82.
          DN get OM certificate from SCM CA for block token validation Sub-task Resolved Ajay Kumar

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 10h 50m
          83.
          Refactor om token db as column family. Sub-task Resolved Anu Engineer  
          84.
          Enable service policy authorization for HDDS group Sub-task Resolved Ajay Kumar  
          85.
          OM get the certificate from SCM CA for token validation Sub-task Resolved Xiaoyu Yao  
          86.
          Change hadoop-runner and apache/hadoop base image to use Java8 Sub-task Resolved Marton Elek

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 0.5h
          87.
          Update MiniOzoneCluster to work with security protocol from SCM Sub-task Resolved Unassigned  
          88.
          Support TokenIssuer interface for running jobs with OzoneFileSystem Sub-task Resolved Xiaoyu Yao

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 6.5h
          89.
          Set OmKeyArgs#refreshPipeline flag properly to avoid reading from stale pipeline Sub-task Resolved Xiaoyu Yao

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          90.
          Refactor ozone acceptance test to allow run in secure mode Sub-task Resolved Ajay Kumar

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 6h 40m
          91.
          Ozone serialization codec for Ozone S3 secret table Sub-task Resolved Zsolt Venczel  
          92.
          KeyOutputStream#write throws ArrayIndexOutOfBoundsException when running RandomWrite MR examples Sub-task Resolved Shashikant Banerjee

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          93.
          Fix MalformedTracerStateStringException on DN logs Sub-task Resolved Xiaoyu Yao

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h
          94.
          Spark job fails to create ozone rpc client Sub-task Resolved Ajay Kumar  
          95.
          NPE if secure ozone if KMS uri is not defined. Sub-task Resolved Ajay Kumar

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          96.
          ozone spark job failing with class not found error for hadoop 2 Sub-task Resolved Ajay Kumar  
          97.
          Inconsistent naming convention with Ozone Kerberos configuration Sub-task Resolved Xiaoyu Yao

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 20m
          98.
          Update ratis dependency to 0.3.0 Sub-task Resolved Ajay Kumar

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          99.
          Add retry to kinit command in smoketests Sub-task Resolved Ajay Kumar

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h

            Activity

              People

              • Assignee:
                xyao Xiaoyu Yao
                Reporter:
                aengineer Anu Engineer
              • Votes:
                0 Vote for this issue
                Watchers:
                26 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 85h 40m
                  85h 40m