Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6134

Transparent data at rest encryption

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0, 3.0.0-alpha1
    • Fix Version/s: 2.6.0
    • Component/s: security
    • Labels:
      None

      Description

      Because of privacy and security regulations, for many industries, sensitive data at rest must be in encrypted form. For example: the health­care industry (HIPAA regulations), the card payment industry (PCI DSS regulations) or the US government (FISMA regulations).

      This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can be used transparently by any application accessing HDFS via Hadoop Filesystem Java API, Hadoop libhdfs C library, or WebHDFS REST API.

      The resulting implementation should be able to be used in compliance with different regulation requirements.

        Attachments

        1. fs-encryption.2014-08-19.patch
          653 kB
          Andrew Wang
        2. fs-encryption.2014-08-18.patch
          653 kB
          Andrew Wang
        3. HDFSDataatRestEncryption.pdf
          357 kB
          Charles Lamb
        4. HDFS-6134.002.patch
          562 kB
          Yi Liu
        5. HDFS-6134.001.patch
          566 kB
          Charles Lamb
        6. HDFS-6134_test_plan.pdf
          147 kB
          Stephen Chu
        7. HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf
          86 kB
          Alejandro Abdelnur
        8. HDFSDataatRestEncryptionProposal_obsolete.pdf
          219 kB
          Alejandro Abdelnur

        Issue Links

        1.
        HDFS Encryption Zones Sub-task Resolved Charles Lamb Actions
        2.
        HDFS integration with KeyProvider Sub-task Resolved Charles Lamb Actions
        3.
        Wire crypto streams for encrypted files in DFSClient Sub-task Resolved Charles Lamb Actions
        4.
        Protocol and API for Encryption Zones Sub-task Resolved Charles Lamb Actions
        5.
        Print out the KeyProvider after finding KP successfully on startup Sub-task Resolved Juan Yu Actions
        6.
        CryptoCode.generateSecureRandom should be a static method Sub-task Resolved Charles Lamb Actions
        7.
        HDFS CLI admin tool for creating & deleting an encryption zone Sub-task Resolved Charles Lamb Actions
        8.
        Get the Key/IV from the NameNode for encrypted files in DFSClient Sub-task Resolved Andrew Wang Actions
        9.
        Rename restrictions for encryption zones Sub-task Resolved Charles Lamb Actions
        10.
        Client server negotiation of cipher suite Sub-task Resolved Andrew Wang Actions
        11.
        Remove the Delete Encryption Zone function Sub-task Resolved Charles Lamb Actions
        12.
        List of Encryption Zones should be based on inodes Sub-task Resolved Charles Lamb Actions
        13.
        Test Crypto streams in HDFS Sub-task Resolved Yi Liu Actions
        14.
        Namenode needs to get the actual keys and iv from the KeyProvider Sub-task Resolved Andrew Wang Actions
        15.
        Clean up encryption-related tests Sub-task Resolved Andrew Wang Actions
        16.
        Fix the keyid format for generated keys in FSNamesystem.createEncryptionZone Sub-task Resolved Charles Lamb Actions
        17.
        Not able to create symlinks after HDFS-6516 Sub-task Resolved Uma Maheswara Rao G Actions
        18.
        Refactor encryption zone functionality into new EncryptionZoneManager class Sub-task Resolved Andrew Wang Actions
        19.
        Update usage of KeyProviderCryptoExtension APIs on NameNode Sub-task Resolved Andrew Wang Actions
        20.
        Remove EncryptionZoneManager lock Sub-task Resolved Andrew Wang Actions
        21.
        Remove unnecessary getEncryptionZoneForPath call in EZManager#createEncryptionZone Sub-task Resolved Uma Maheswara Rao G Actions
        22.
        Remove KeyProvider in EncryptionZoneManager Sub-task Resolved Andrew Wang Actions
        23.
        Decrypt EDEK before creating CryptoInputStream/CryptoOutputStream Sub-task Resolved Andrew Wang Actions
        24.
        Creating encryption zone results in NPE when KeyProvider is null Sub-task Resolved Charles Lamb Actions
        25.
        Create a special /.reserved/raw directory for raw access to encrypted data Sub-task Resolved Charles Lamb Actions
        26.
        Create a .RAW extended attribute namespace Sub-task Resolved Charles Lamb Actions
        27.
        Add more HDFS encryption tests Sub-task Resolved Andrew Wang Actions
        28.
        Should not be able to create encryption zone using path to a non-directory file Sub-task Resolved Charles Lamb Actions
        29.
        Require specification of an encryption key when creating an encryption zone Sub-task Resolved Andrew Wang Actions
        30.
        Batch the encryption zones listing API Sub-task Resolved Andrew Wang Actions
        31.
        DFSClient should use IV generated based on the configured CipherSuite with codecs used Sub-task Resolved Uma Maheswara Rao G Actions
        32.
        Cannot remove directory within encryption zone to Trash Sub-task Resolved Unassigned Actions
        33.
        Fix TestReservedRawPaths failures Sub-task Resolved Charles Lamb Actions
        34.
        Mistakenly dfs.namenode.list.encryption.zones.num.responses configured as boolean Sub-task Resolved Uma Maheswara Rao G Actions
        35.
        HDFS encryption documentation Sub-task Resolved Andrew Wang Actions
        36.
        Fix findbugs and other warnings Sub-task Resolved Yi Liu Actions
        37.
        Improve the configuration guidance in DFSClient when there are no Codec classes found in configs Sub-task Resolved Uma Maheswara Rao G Actions
        38.
        Fix TestCLI to expect new output Sub-task Resolved Charles Lamb Actions
        39.
        Add non-superuser capability to get the encryption zone for a specific path Sub-task Resolved Charles Lamb Actions
        40.
        Constants in CommandWithDestination should be static Sub-task Resolved Charles Lamb Actions

          Activity

            People

            • Assignee:
              clamb Charles Lamb
              Reporter:
              tucu00 Alejandro Abdelnur

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment