[HDFS-2856] Fix block protocol so that Datanodes don't require root or jsvc

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0, 3.0.0-alpha1
• Fix Version/s: 2.6.0
• Component/s: datanode, security
• Target Version/s: 2.6.0
• Hadoop Flags: Reviewed

Release Note

SASL now can be used to secure the DataTransferProtocol, which transfers file block content between HDFS clients and DataNodes. In this configuration, it is no longer required for secured clusters to start the DataNode as root and bind to privileged ports.

• Assignee: Chris Nauroth
• Reporter: Owen O'Malley
• Created: 29/Jan/12 18:12
• Updated: 12/May/16 18:12
• Resolved: 14/Jul/14 18:29

Description

Since we send the block tokens unencrypted to the datanode, we currently start the datanode as root using jsvc and get a secure (< 1024) port.

If we have the datanode generate a nonce and send it on the connection and the sends an hmac of the nonce back instead of the block token it won't reveal any secrets. Thus, we wouldn't require a secure port and would not require root or jsvc.

Issue Links

• breaks: HDFS-7382 DataNode in secure mode may throw NullPointerException if client connects before DataNode registers itself with NameNode.
• is related to: HDFS-6600 fsck -move fails in secured clusters.
• is related to: HDFS-3749 Disable check for jsvc on windows
• is related to: HDFS-7073 Allow falling back to a non-SASL connection on DataTransferProtocol in several edge cases.
• is related to: HDFS-7367 HDFS short-circuit read cannot negotiate shared memory slot and file descriptors when SASL is enabled on DataTransferProtocol.
• is related to: HDFS-6859 Allow dfs.data.transfer.protection default to hadoop.rpc.protection
• is related to: HDFS-6606 Optimize HDFS Encrypted Transport performance
• is related to: HDFS-7195 Update user doc of secure mode about Datanodes don't require root or jsvc
• is related to: HDFS-6858 Allow dfs.data.transfer.saslproperties.resolver.class default to hadoop.security.saslproperties.resolver.class
• is related to: HDFS-7386 Replace check "port number < 1024" with shared isPrivilegedPort method
• relates to: HDFS-3637 Add support for encrypting the DataTransferProtocol
• relates to: HADOOP-9671 Improve Hadoop security - Use cases, Threat Model and Problems