Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13541

NameNode Port based selective encryption

    XMLWordPrintableJSON

Details

    • Hide
      This feature allows HDFS to selectively enforce encryption for both RPC (NameNode) and data transfer (DataNode). With this feature enabled, NameNode can listen on multiple ports, and different ports can have different security configurations. Depending on which NameNode port clients connect to, the RPC calls and the following data transfer will enforce security configuration corresponding to this NameNode port. This can help when there is requirement to enforce different security policies depending on the location where the clients are connecting from.

      This can be enabled by setting `hadoop.security.saslproperties.resolver.class` configuration to `org.apache.hadoop.security.IngressPortBasedResolver`, and add the additional NameNode auxiliary ports by setting `dfs.namenode.rpc-address.auxiliary-ports`, and set the security individual ports by configuring `ingress.port.sasl.configured.ports`.
      Show
      This feature allows HDFS to selectively enforce encryption for both RPC (NameNode) and data transfer (DataNode). With this feature enabled, NameNode can listen on multiple ports, and different ports can have different security configurations. Depending on which NameNode port clients connect to, the RPC calls and the following data transfer will enforce security configuration corresponding to this NameNode port. This can help when there is requirement to enforce different security policies depending on the location where the clients are connecting from. This can be enabled by setting `hadoop.security.saslproperties.resolver.class` configuration to `org.apache.hadoop.security.IngressPortBasedResolver`, and add the additional NameNode auxiliary ports by setting `dfs.namenode.rpc-address.auxiliary-ports`, and set the security individual ports by configuring `ingress.port.sasl.configured.ports`.

    Description

      Here at LinkedIn, one issue we face is that we need to enforce different security requirement based on the location of client and the cluster. Specifically, for clients from outside of the data center, it is required by regulation that all traffic must be encrypted. But for clients within the same data center, unencrypted connections are more desired to avoid the high encryption overhead. 

      HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335 introduced WhitelistBasedResolver which solves the same problem. However we found it difficult to fit into our environment for several reasons. In this JIRA, on top of pluggable SASL resolver, we propose a different approach of running RPC two ports on NameNode, and the two ports will be enforcing encrypted and unencrypted connections respectively, and the following DataNode access will simply follow the same behaviour of encryption/unencryption. Then by blocking unencrypted port on datacenter firewall, we can completely block unencrypted external access.

      Attachments

        1. HDFS-13541-branch-2.001.patch
          107 kB
          Chen Liang
        2. HDFS-13541-branch-2.002.patch
          107 kB
          Chen Liang
        3. HDFS-13541-branch-2.003.patch
          105 kB
          Chen Liang
        4. HDFS-13541-branch-3.1.001.patch
          107 kB
          Chen Liang
        5. HDFS-13541-branch-3.1.002.patch
          107 kB
          Chen Liang
        6. HDFS-13541-branch-3.2.001.patch
          96 kB
          Chen Liang
        7. HDFS-13541-branch-3.2.002.patch
          96 kB
          Chen Liang
        8. NameNode Port based selective encryption-v1.pdf
          133 kB
          Chen Liang

        Issue Links

          Activity

            People

              vagarychen Chen Liang
              vagarychen Chen Liang
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: