Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6134 Transparent data at rest encryption
  3. HDFS-6737

DFSClient should use IV generated based on the configured CipherSuite with codecs used

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: fs-encryption (HADOOP-10150 and HDFS-6134)
    • Fix Version/s: None
    • Component/s: hdfs-client
    • Labels:
      None

      Description

      Seems like we are using IV as like Encrypted data encryption key iv. But the underlying Codec's cipher suite may expect different iv length. So, we should generate IV from the Coec's cipher suite configured.

       final CryptoInputStream cryptoIn =
                new CryptoInputStream(dfsis, CryptoCodec.getInstance(conf, 
                    feInfo.getCipherSuite()), feInfo.getEncryptedDataEncryptionKey(),
                    feInfo.getIV());
      

      So, instead of using feinfo.getIV(), we should generate like

      byte[] iv = new byte[codec.getCipherSuite().getAlgorithmBlockSize()]; 
      codec.generateSecureRandom(iv);
      
      1. HDFS-6737.patch
        0.9 kB
        Uma Maheswara Rao G

        Activity

        Hide
        Uma Maheswara Rao G added a comment -

        I realized while changing the code that, creating the instance with feinfo.getCipherSuite only. So, the cipherSuite what we used should expect the same length as fsInfo.getIV.
        So, But in FsNamesystem#startFileInternal generating the IV with edek, but do we need to generate the IV as the length of passed suite algorithm block size? If so, I will update JIRA description and post the patch for it. Please correct me if I did not follow.

        Show
        Uma Maheswara Rao G added a comment - I realized while changing the code that, creating the instance with feinfo.getCipherSuite only. So, the cipherSuite what we used should expect the same length as fsInfo.getIV. So, But in FsNamesystem#startFileInternal generating the IV with edek, but do we need to generate the IV as the length of passed suite algorithm block size? If so, I will update JIRA description and post the patch for it. Please correct me if I did not follow.
        Hide
        Uma Maheswara Rao G added a comment -

        Attaching a patch as per my comment above to make it clear.

        Show
        Uma Maheswara Rao G added a comment - Attaching a patch as per my comment above to make it clear.
        Hide
        Andrew Wang added a comment -

        Hi Uma, good points here. I chatted with Alejandro Abdelnur about this, here's how it works right now:

        • Each encryption zone has an ezKey of some size. generateEncryptedKey hardcodes usage of AES/CTR/NoPadding, which means a 16B IV.
        • When generating a new encrypted key, it has the same keySize as the ezKey and same IV size as the hardcoded AES/CTR/NoPadding
        • All AES algorithms uses a 16B IV, so we're find as long as the DEK is always AES too (okay limitation)
        • We don't foresee switching the hardcoded AES/CTR/NoPadding, so don't need to pass a CipherSuite into generate/decryptEncryptedKey
        • Enforcing that the ezKey and DEK need to have the same keySize is not great, but tucu thinks it's a reasonable limitation. If a user wants to change the keysize, they need to make a new EZ with a bigger ezKey and copy everything there.
        • You can still use whatever AES algorithm you want for the actual data encryption, which is what the per-file CipherSuite specifies.

        I find this pretty complicated, so is definitely something we need to put in the user documentation. createEncryptionZone also seems like it needs a way of specifying the key size, but we could do that when we actually support AES-256. Do you think we need any other improvements? We could try to improve how things are modeled in CipherSuite (since we depend on the block size being 16B), but maybe it's okay as is.

        Show
        Andrew Wang added a comment - Hi Uma, good points here. I chatted with Alejandro Abdelnur about this, here's how it works right now: Each encryption zone has an ezKey of some size. generateEncryptedKey hardcodes usage of AES/CTR/NoPadding, which means a 16B IV. When generating a new encrypted key, it has the same keySize as the ezKey and same IV size as the hardcoded AES/CTR/NoPadding All AES algorithms uses a 16B IV, so we're find as long as the DEK is always AES too (okay limitation) We don't foresee switching the hardcoded AES/CTR/NoPadding, so don't need to pass a CipherSuite into generate/decryptEncryptedKey Enforcing that the ezKey and DEK need to have the same keySize is not great, but tucu thinks it's a reasonable limitation. If a user wants to change the keysize, they need to make a new EZ with a bigger ezKey and copy everything there. You can still use whatever AES algorithm you want for the actual data encryption, which is what the per-file CipherSuite specifies. I find this pretty complicated, so is definitely something we need to put in the user documentation. createEncryptionZone also seems like it needs a way of specifying the key size, but we could do that when we actually support AES-256. Do you think we need any other improvements? We could try to improve how things are modeled in CipherSuite (since we depend on the block size being 16B), but maybe it's okay as is.
        Hide
        Uma Maheswara Rao G added a comment -

        Ok. Thanks Andrew Wang for the discussion with Alejandro Abdelnur about this. Then on the decision of we support only AES/CTR/NoPadding mode, this is ok to leave as is. I will close this as won't fix.

        Show
        Uma Maheswara Rao G added a comment - Ok. Thanks Andrew Wang for the discussion with Alejandro Abdelnur about this. Then on the decision of we support only AES/CTR/NoPadding mode, this is ok to leave as is. I will close this as won't fix.

          People

          • Assignee:
            Uma Maheswara Rao G
            Reporter:
            Uma Maheswara Rao G
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development