Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6134

Transparent data at rest encryption

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.3.0, 3.0.0-alpha1
    • 2.6.0
    • security
    • None

    Description

      Because of privacy and security regulations, for many industries, sensitive data at rest must be in encrypted form. For example: the health­care industry (HIPAA regulations), the card payment industry (PCI DSS regulations) or the US government (FISMA regulations).

      This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can be used transparently by any application accessing HDFS via Hadoop Filesystem Java API, Hadoop libhdfs C library, or WebHDFS REST API.

      The resulting implementation should be able to be used in compliance with different regulation requirements.

      Attachments

        1. fs-encryption.2014-08-18.patch
          653 kB
          Andrew Wang
        2. fs-encryption.2014-08-19.patch
          653 kB
          Andrew Wang
        3. HDFS-6134_test_plan.pdf
          147 kB
          Stephen Chu
        4. HDFS-6134.001.patch
          566 kB
          Charles Lamb
        5. HDFS-6134.002.patch
          562 kB
          Yi Liu
        6. HDFSDataatRestEncryption.pdf
          357 kB
          Charles Lamb
        7. HDFSDataatRestEncryptionProposal_obsolete.pdf
          219 kB
          Alejandro Abdelnur
        8. HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf
          86 kB
          Alejandro Abdelnur

        Issue Links

          1.
          HDFS Encryption Zones Sub-task Resolved Charles Lamb
          2.
          HDFS integration with KeyProvider Sub-task Resolved Charles Lamb
          3.
          Wire crypto streams for encrypted files in DFSClient Sub-task Resolved Charles Lamb
          4.
          Protocol and API for Encryption Zones Sub-task Resolved Charles Lamb
          5.
          Print out the KeyProvider after finding KP successfully on startup Sub-task Resolved Juan Yu
          6.
          CryptoCode.generateSecureRandom should be a static method Sub-task Resolved Charles Lamb
          7.
          HDFS CLI admin tool for creating & deleting an encryption zone Sub-task Resolved Charles Lamb
          8.
          Get the Key/IV from the NameNode for encrypted files in DFSClient Sub-task Resolved Andrew Wang
          9.
          Rename restrictions for encryption zones Sub-task Resolved Charles Lamb
          10.
          Client server negotiation of cipher suite Sub-task Resolved Andrew Wang
          11.
          Remove the Delete Encryption Zone function Sub-task Resolved Charles Lamb
          12.
          List of Encryption Zones should be based on inodes Sub-task Resolved Charles Lamb
          13.
          Test Crypto streams in HDFS Sub-task Resolved Yi Liu
          14.
          Namenode needs to get the actual keys and iv from the KeyProvider Sub-task Resolved Andrew Wang
          15.
          Clean up encryption-related tests Sub-task Resolved Andrew Wang
          16.
          Fix the keyid format for generated keys in FSNamesystem.createEncryptionZone Sub-task Resolved Charles Lamb
          17.
          Not able to create symlinks after HDFS-6516 Sub-task Resolved Uma Maheswara Rao G
          18.
          Refactor encryption zone functionality into new EncryptionZoneManager class Sub-task Resolved Andrew Wang
          19.
          Update usage of KeyProviderCryptoExtension APIs on NameNode Sub-task Resolved Andrew Wang
          20.
          Remove EncryptionZoneManager lock Sub-task Resolved Andrew Wang
          21.
          Remove unnecessary getEncryptionZoneForPath call in EZManager#createEncryptionZone Sub-task Resolved Uma Maheswara Rao G
          22.
          Remove KeyProvider in EncryptionZoneManager Sub-task Resolved Andrew Wang
          23.
          Decrypt EDEK before creating CryptoInputStream/CryptoOutputStream Sub-task Resolved Andrew Wang
          24.
          Creating encryption zone results in NPE when KeyProvider is null Sub-task Resolved Charles Lamb
          25.
          Create a special /.reserved/raw directory for raw access to encrypted data Sub-task Resolved Charles Lamb
          26.
          Create a .RAW extended attribute namespace Sub-task Resolved Charles Lamb
          27.
          Add more HDFS encryption tests Sub-task Resolved Andrew Wang
          28.
          Should not be able to create encryption zone using path to a non-directory file Sub-task Resolved Charles Lamb
          29.
          Require specification of an encryption key when creating an encryption zone Sub-task Resolved Andrew Wang
          30.
          Batch the encryption zones listing API Sub-task Resolved Andrew Wang
          31.
          DFSClient should use IV generated based on the configured CipherSuite with codecs used Sub-task Resolved Uma Maheswara Rao G
          32.
          Cannot remove directory within encryption zone to Trash Sub-task Resolved Unassigned
          33.
          Fix TestReservedRawPaths failures Sub-task Resolved Charles Lamb
          34.
          Mistakenly dfs.namenode.list.encryption.zones.num.responses configured as boolean Sub-task Resolved Uma Maheswara Rao G
          35.
          HDFS encryption documentation Sub-task Resolved Andrew Wang
          36.
          Fix findbugs and other warnings Sub-task Resolved Yi Liu
          37.
          Improve the configuration guidance in DFSClient when there are no Codec classes found in configs Sub-task Resolved Uma Maheswara Rao G
          38.
          Fix TestCLI to expect new output Sub-task Resolved Charles Lamb
          39.
          Add non-superuser capability to get the encryption zone for a specific path Sub-task Resolved Charles Lamb
          40.
          Constants in CommandWithDestination should be static Sub-task Resolved Charles Lamb

          Activity

            People

              clamb Charles Lamb
              tucu00 Alejandro Abdelnur
              Votes:
              2 Vote for this issue
              Watchers:
              62 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: