Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 2.3.0
    • Fix Version/s: 2.6.0
    • Component/s: security
    • Labels:
      None

      Description

      Because of privacy and security regulations, for many industries, sensitive data at rest must be in encrypted form. For example: the health­care industry (HIPAA regulations), the card payment industry (PCI DSS regulations) or the US government (FISMA regulations).

      This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can be used transparently by any application accessing HDFS via Hadoop Filesystem Java API, Hadoop libhdfs C library, or WebHDFS REST API.

      The resulting implementation should be able to be used in compliance with different regulation requirements.

      1. HDFSDataatRestEncryptionProposal_obsolete.pdf
        219 kB
        Alejandro Abdelnur
      2. HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf
        86 kB
        Alejandro Abdelnur
      3. HDFS-6134_test_plan.pdf
        147 kB
        Stephen Chu
      4. HDFS-6134.001.patch
        566 kB
        Charles Lamb
      5. HDFS-6134.002.patch
        562 kB
        Yi Liu
      6. HDFSDataatRestEncryption.pdf
        357 kB
        Charles Lamb
      7. fs-encryption.2014-08-18.patch
        653 kB
        Andrew Wang
      8. fs-encryption.2014-08-19.patch
        653 kB
        Andrew Wang

        Issue Links

        1.
        HDFS Encryption Zones Sub-task Resolved Charles Lamb
         
        2.
        HDFS integration with KeyProvider Sub-task Resolved Charles Lamb
         
        3.
        Wire crypto streams for encrypted files in DFSClient Sub-task Resolved Charles Lamb
         
        4.
        Protocol and API for Encryption Zones Sub-task Resolved Charles Lamb
         
        5.
        Print out the KeyProvider after finding KP successfully on startup Sub-task Resolved Juan Yu
         
        6.
        CryptoCode.generateSecureRandom should be a static method Sub-task Resolved Charles Lamb
         
        7.
        HDFS CLI admin tool for creating & deleting an encryption zone Sub-task Resolved Charles Lamb
         
        8.
        Get the Key/IV from the NameNode for encrypted files in DFSClient Sub-task Resolved Andrew Wang
         
        9.
        Rename restrictions for encryption zones Sub-task Resolved Charles Lamb
         
        10.
        Client server negotiation of cipher suite Sub-task Resolved Andrew Wang
         
        11.
        Remove the Delete Encryption Zone function Sub-task Resolved Charles Lamb
         
        12.
        List of Encryption Zones should be based on inodes Sub-task Resolved Charles Lamb
         
        13.
        Test Crypto streams in HDFS Sub-task Resolved Yi Liu
         
        14.
        Namenode needs to get the actual keys and iv from the KeyProvider Sub-task Resolved Andrew Wang
         
        15.
        Clean up encryption-related tests Sub-task Resolved Andrew Wang
         
        16.
        Fix the keyid format for generated keys in FSNamesystem.createEncryptionZone Sub-task Resolved Charles Lamb
         
        17.
        Not able to create symlinks after HDFS-6516 Sub-task Resolved Uma Maheswara Rao G
         
        18.
        Refactor encryption zone functionality into new EncryptionZoneManager class Sub-task Resolved Andrew Wang
         
        19.
        Update usage of KeyProviderCryptoExtension APIs on NameNode Sub-task Resolved Andrew Wang
         
        20.
        Remove EncryptionZoneManager lock Sub-task Resolved Andrew Wang
         
        21.
        Remove unnecessary getEncryptionZoneForPath call in EZManager#createEncryptionZone Sub-task Resolved Uma Maheswara Rao G
         
        22.
        Remove KeyProvider in EncryptionZoneManager Sub-task Resolved Andrew Wang
         
        23.
        Decrypt EDEK before creating CryptoInputStream/CryptoOutputStream Sub-task Resolved Andrew Wang
         
        24.
        Creating encryption zone results in NPE when KeyProvider is null Sub-task Resolved Charles Lamb
         
        25.
        Create a special /.reserved/raw directory for raw access to encrypted data Sub-task Resolved Charles Lamb
         
        26.
        Create a .RAW extended attribute namespace Sub-task Resolved Charles Lamb
         
        27.
        Add more HDFS encryption tests Sub-task Resolved Andrew Wang
         
        28.
        Should not be able to create encryption zone using path to a non-directory file Sub-task Resolved Charles Lamb
         
        29.
        Require specification of an encryption key when creating an encryption zone Sub-task Resolved Andrew Wang
         
        30.
        Batch the encryption zones listing API Sub-task Resolved Andrew Wang
         
        31.
        DFSClient should use IV generated based on the configured CipherSuite with codecs used Sub-task Resolved Uma Maheswara Rao G
         
        32.
        Cannot remove directory within encryption zone to Trash Sub-task Resolved Unassigned
         
        33.
        Fix TestReservedRawPaths failures Sub-task Resolved Charles Lamb
         
        34.
        Mistakenly dfs.namenode.list.encryption.zones.num.responses configured as boolean Sub-task Resolved Uma Maheswara Rao G
         
        35.
        HDFS encryption documentation Sub-task Resolved Andrew Wang
         
        36.
        Fix findbugs and other warnings Sub-task Resolved Yi Liu
         
        37.
        Improve the configuration guidance in DFSClient when there are no Codec classes found in configs Sub-task Resolved Uma Maheswara Rao G
         
        38.
        Fix TestCLI to expect new output Sub-task Resolved Charles Lamb
         
        39.
        Add non-superuser capability to get the encryption zone for a specific path Sub-task Resolved Charles Lamb
         
        40.
        Constants in CommandWithDestination should be static Sub-task Resolved Charles Lamb
         

          Activity

            People

            • Assignee:
              Charles Lamb
              Reporter:
              Alejandro Abdelnur
            • Votes:
              2 Vote for this issue
              Watchers:
              58 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development