Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10150

Hadoop cryptographic file system

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.0.0-alpha1
    • 2.6.0
    • security

    Description

      There is an increasing need for securing data when Hadoop customers use various upper layer applications, such as Map-Reduce, Hive, Pig, HBase and so on.

      HADOOP CFS (HADOOP Cryptographic File System) is used to secure data, based on HADOOP “FilterFileSystem” decorating DFS or other file systems, and transparent to upper layer applications. It’s configurable, scalable and fast.

      High level requirements:
      1. Transparent to and no modification required for upper layer applications.
      2. “Seek”, “PositionedReadable” are supported for input stream of CFS if the wrapped file system supports them.
      3. Very high performance for encryption and decryption, they will not become bottleneck.
      4. Can decorate HDFS and all other file systems in Hadoop, and will not modify existing structure of file system, such as namenode and datanode structure if the wrapped file system is HDFS.
      5. Admin can configure encryption policies, such as which directory will be encrypted.
      6. A robust key management framework.
      7. Support Pread and append operations if the wrapped file system supports them.

      Attachments

        1. cfs.patch
          104 kB
          Yi Liu
        2. CryptographicFileSystem.patch
          287 kB
          Yi Liu
        3. extended information based on INode feature.patch
          128 kB
          Yi Liu
        4. HADOOP cryptographic file system.pdf
          561 kB
          Yi Liu
        5. HADOOP cryptographic file system-V2.docx
          103 kB
          Yi Liu
        6. HDFSDataAtRestEncryptionAlternatives.pdf
          321 kB
          Alejandro Abdelnur
        7. HDFSDataatRestEncryptionAttackVectors.pdf
          131 kB
          Alejandro Abdelnur
        8. HDFSDataatRestEncryptionProposal.pdf
          219 kB
          Alejandro Abdelnur

        Issue Links

          1.
          Crypto input and output streams implementing Hadoop stream interfaces Sub-task Resolved Yi Liu
          2.
          Tests for Crypto input and output streams using fake streams implementing Hadoop streams interfaces. Sub-task Resolved Yi Liu
          3.
          Javadoc and few code style improvement for Crypto input and output streams Sub-task Resolved Yi Liu
          4.
          Minor improvements to Crypto input and output streams Sub-task Closed Yi Liu
          5.
          Add a method to CryptoCodec to generate SRNs for IV Sub-task Closed Yi Liu
          6.
          Add a new constructor for CryptoInputStream that receives current position of wrapped stream. Sub-task Resolved Yi Liu
          7.
          NullPointerException in CryptoInputStream while wrapped stream is not ByteBufferReadable. Add tests using normal stream. Sub-task Resolved Yi Liu
          8.
          Implementation of AES-CTR CryptoCodec using JNI to OpenSSL Sub-task Resolved Yi Liu
          9.
          Refactor CryptoCodec#generateSecureRandom to take a byte[] Sub-task Resolved Andrew Wang
          10.
          Implement high-performance secure random number sources Sub-task Resolved Yi Liu
          11.
          Fall back AesCtrCryptoCodec implementation from OpenSSL to JCE if non native support. Sub-task Resolved Yi Liu
          12.
          UnsatisfiedLinkError in cryptocodec tests with OpensslCipher#initContext Sub-task Resolved Uma Maheswara Rao G
          13.
          Update OpensslCipher#getInstance to accept CipherSuite#name format. Sub-task Resolved Yi Liu
          14.
          Refactor get instance of CryptoCodec and support create via algorithm/mode/padding. Sub-task Resolved Yi Liu
          15.
          Failed to load OpenSSL cipher error logs on systems with old openssl versions Sub-task Resolved Colin McCabe
          16.
          incorrect prototype in OpensslSecureRandom.c Sub-task Resolved Colin McCabe
          17.
          CryptoCodec#getCodecclasses throws NPE when configurations not loaded. Sub-task Closed Uma Maheswara Rao G

          Activity

            People

              hitliuyi Yi Liu
              hitliuyi Yi Liu
              Votes:
              0 Vote for this issue
              Watchers:
              54 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: