Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
SASL secured connections are commonly configured to negotiate confidential (encrypted) connections, known as the "auth-conf" quality of protection. This ensures both authentication and data encryption, enhancing the security of wire communication. The use of AES encryption, negotiated on "auth-conf" connections with Kerberos/GSSAPI, meets the requirements of modern commercial and governmental cryptographic regulations and policies.
However, when deploying a YARN job that incorporates a network client expecting to negotiate the same level of security (for example an HBase client, but any code that integrates Hadoop's UGI and related and the JRE's SASLClient will be affected). The problem arises from the fact that delegation tokens, the only hard-coded option available for tasks, rely on the Digest-MD5 SASL mechanism. Unfortunately, the Digest-MD5 negotiation standard supports only five outdated and slow ciphers for SASL confidentiality: RC4 (40 bits key length), RC4 (56 bits key length), RC4 (128 bits key length), DES, and Triple DES. Notably, the use of RC4 has been prohibited by the IETF since 2015, and DES was compromised in 1999 and subsequently withdrawn as a standard by NIST.
The limitations of the Digest-MD5 mechanism have significant implications for compliance with modern cryptographic regulations and policies that mandate wire encryption. As a result, YARN applications utilizing Digest-MD5 for confidentiality negotiation cannot adhere to these requirements. It is worth noting that this issue is not documented in the Hadoop documentation or logs, potentially leading developers and operators to remain unaware of the problem.
How does hadoop delgation token works - https://blog.cloudera.com/hadoop-delegation-tokens-explained/
Hadoop Security Design - http://hortonworks.com/wp-content/uploads/2011/08/adding_security_to_apache_hadoop.pdf