Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15776

The key length used in "KeyGenerator.init()" should be configurable

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: conf
    • Labels:
      None

      Description

      In mapreduce, the key length used in "KeyGenerator.init()" is configured with configuration option "mapreduce.job.encrypted-intermediate-data-key-size-bits" as follows:

       

      /org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
      protected void initJobCredentialsAndUGI(Configuration conf) {
        ...
        int keyLen = conf.getInt(
                      MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA_KEY_SIZE_BITS,
                      MRJobConfig
                              .DEFAULT_MR_ENCRYPTED_INTERMEDIATE_DATA_KEY_SIZE_BITS);
        KeyGenerator keyGen =
                      KeyGenerator.getInstance(INTERMEDIATE_DATA_ENCRYPTION_ALGO);
        keyGen.init(keyLen);
        encryptedSpillKey = keyGen.generateKey().getEncoded();
        ...
      }
      

      The same usage is also in mapred as follows:

      /org/apache/hadoop/mapred/LocalJobRunner.java
      public Job(JobID jobid, String jobSubmitDir) throws IOException {  ...
        int keyLen = conf.getInt(
                      MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA_KEY_SIZE_BITS,
                      MRJobConfig
                              .DEFAULT_MR_ENCRYPTED_INTERMEDIATE_DATA_KEY_SIZE_BITS);
        KeyGenerator keyGen =
                      KeyGenerator.getInstance(INTERMEDIATE_DATA_ENCRYPTION_ALGO);
        keyGen.init(keyLen);
        ...
      }
      

      Also, in hadoop-common, there is a configration option "hadoop.security.key.default.bitlength", it is used in "KeyProvider.java" to initiate KeyGenerator as follows:

      /org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
      public static final String HADOOP_SECURITY_KEY_DEFAULT_BITLENGTH_KEY =
            "hadoop.security.key.default.bitlength";
        /** Defalt value for HADOOP_SECURITY_KEY_DEFAULT_BITLENGTH_KEY. */
        public static final int HADOOP_SECURITY_KEY_DEFAULT_BITLENGTH_DEFAULT = 128;
      
      /org/apache/hadoop/crypto/key/KeyProvider.java
      public Options(Configuration conf) {
        cipher = conf.get(DEFAULT_CIPHER_NAME, DEFAULT_CIPHER);
        bitLength = conf.getInt(DEFAULT_BITLENGTH_NAME, DEFAULT_BITLENGTH);
      }
      
      public KeyVersion createKey(String name, Options options)
            throws NoSuchAlgorithmException, IOException {
          byte[] material = generateKey(options.getBitLength(), options.getCipher());
          return createKey(name, material, options);
        }
      
      protected byte[] generateKey(int size, String algorithm)
            throws NoSuchAlgorithmException {
          algorithm = getAlgorithm(algorithm);
          KeyGenerator keyGenerator = KeyGenerator.getInstance(algorithm);
          keyGenerator.init(size);
          byte[] key = keyGenerator.generateKey().getEncoded();
          return key;
        }
        ...
      }
      

      However, in other two usage of "KeyGenerator.init()" in mapreduce and hadoop-common, the key length is hard-coded as 64. Also, in the evolving history, this value is changed from "20" to "64". So, in the perspective of flexibility and security, these two hard coded value in "KeyGenerator.init()" should be configurable.

      /org/apache/hadoop/mapreduce/JobSubmitter.java
      class JobSubmitter {
        ...
        private static final int SHUFFLE_KEY_LENGTH = 64;
        ...
        JobStatus submitJobInternal(Job job, Cluster cluster) 
          throws ClassNotFoundException, InterruptedException, IOException {
          ...
          keyGen = KeyGenerator.getInstance(SHUFFLE_KEYGEN_ALGORITHM);
          keyGen.init(SHUFFLE_KEY_LENGTH);
          ...
        }
        ...
      }
      
      /org/apache/hadoop/security/token/SecretManager.java
      public abstract class SecretManager<T extends TokenIdentifier> {
        ...
        private static final int KEY_LENGTH = 64;
        ...
        private final KeyGenerator keyGen;
        {
          try {
            keyGen = KeyGenerator.getInstance(DEFAULT_HMAC_ALGORITHM);
            keyGen.init(KEY_LENGTH);
          } catch (NoSuchAlgorithmException nsa) {
            throw new IllegalArgumentException("Can't find " + DEFAULT_HMAC_ALGORITHM +
            " algorithm.");
          }
        }
        ...
      }
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              freebigshow zhoushulin
            • Votes:
              2 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: