Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 2.6.0
    • Component/s: security
    • Labels:

      Description

      There is an increasing need for securing data when Hadoop customers use various upper layer applications, such as Map-Reduce, Hive, Pig, HBase and so on.

      HADOOP CFS (HADOOP Cryptographic File System) is used to secure data, based on HADOOP “FilterFileSystem” decorating DFS or other file systems, and transparent to upper layer applications. It’s configurable, scalable and fast.

      High level requirements:
      1. Transparent to and no modification required for upper layer applications.
      2. “Seek”, “PositionedReadable” are supported for input stream of CFS if the wrapped file system supports them.
      3. Very high performance for encryption and decryption, they will not become bottleneck.
      4. Can decorate HDFS and all other file systems in Hadoop, and will not modify existing structure of file system, such as namenode and datanode structure if the wrapped file system is HDFS.
      5. Admin can configure encryption policies, such as which directory will be encrypted.
      6. A robust key management framework.
      7. Support Pread and append operations if the wrapped file system supports them.

      1. HADOOP cryptographic file system.pdf
        561 kB
        Yi Liu
      2. CryptographicFileSystem.patch
        287 kB
        Yi Liu
      3. HADOOP cryptographic file system-V2.docx
        103 kB
        Yi Liu
      4. extended information based on INode feature.patch
        128 kB
        Yi Liu
      5. cfs.patch
        104 kB
        Yi Liu
      6. HDFSDataAtRestEncryptionAlternatives.pdf
        321 kB
        Alejandro Abdelnur
      7. HDFSDataatRestEncryptionProposal.pdf
        219 kB
        Alejandro Abdelnur
      8. HDFSDataatRestEncryptionAttackVectors.pdf
        131 kB
        Alejandro Abdelnur

        Issue Links

        1.
        Crypto input and output streams implementing Hadoop stream interfaces Sub-task Resolved Yi Liu
         
        2.
        Tests for Crypto input and output streams using fake streams implementing Hadoop streams interfaces. Sub-task Resolved Yi Liu
         
        3.
        Javadoc and few code style improvement for Crypto input and output streams Sub-task Resolved Yi Liu
         
        4.
        Minor improvements to Crypto input and output streams Sub-task Closed Yi Liu
         
        5.
        Add a method to CryptoCodec to generate SRNs for IV Sub-task Closed Yi Liu
         
        6.
        Add a new constructor for CryptoInputStream that receives current position of wrapped stream. Sub-task Resolved Yi Liu
         
        7.
        NullPointerException in CryptoInputStream while wrapped stream is not ByteBufferReadable. Add tests using normal stream. Sub-task Resolved Yi Liu
         
        8.
        Implementation of AES-CTR CryptoCodec using JNI to OpenSSL Sub-task Resolved Yi Liu
         
        9.
        Refactor CryptoCodec#generateSecureRandom to take a byte[] Sub-task Resolved Andrew Wang
         
        10.
        Implement high-performance secure random number sources Sub-task Resolved Yi Liu
         
        11.
        Fall back AesCtrCryptoCodec implementation from OpenSSL to JCE if non native support. Sub-task Resolved Yi Liu
         
        12.
        UnsatisfiedLinkError in cryptocodec tests with OpensslCipher#initContext Sub-task Resolved Uma Maheswara Rao G
         
        13.
        Update OpensslCipher#getInstance to accept CipherSuite#name format. Sub-task Resolved Yi Liu
         
        14.
        Refactor get instance of CryptoCodec and support create via algorithm/mode/padding. Sub-task Resolved Yi Liu
         
        15.
        Failed to load OpenSSL cipher error logs on systems with old openssl versions Sub-task Resolved Colin Patrick McCabe
         
        16.
        incorrect prototype in OpensslSecureRandom.c Sub-task Resolved Colin Patrick McCabe
         
        17.
        CryptoCodec#getCodecclasses throws NPE when configurations not loaded. Sub-task Closed Uma Maheswara Rao G
         

          Activity

            People

            • Assignee:
              Yi Liu
              Reporter:
              Yi Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              60 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development