Security: Need to be able to encrypt Hadoop socket connections

• Key: HADOOP-2239
• Type: Improvement
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Chris Douglas
• Reporter: Allen Wittenauer
• Fix Version/s: 0.17.0

Description

We need to be able to use hadoop over hostile networks, both internally and externally to the enterpise. While authentication prevents unauthorized access, encryption should be used to prevent such things as packet snooping across the wire. This means that hadoop client connections, distcp, etc, would use something such as SSL to protect the TCP/IP packets. Post-Kerberos, it would be useful to use something similar to NFS's krb5p option.

Comments

[Owen O'Malley]

More precisely, what we need is the hftp file system to optionally go through ssl.

[Doug Cutting]

This can be triggered when an "hsftp:" uri is used.

The following would be required to implement this:

1. Extend the namenode and datanode's http servers to respond to https connections, adding new configuration properties to name https ports, and configuring jetty accordingly (adding a Jetty SslListener in our StatusHttpServer, specifying certs, etc.).
2. Change HftpFileSystem to use https connections when the "hsftp" scheme is used.
3. Change FileDataServlet to redirect to https when https is used to access it.

Does that sound right?

[Allen Wittenauer]

While we really need secure distcp Right Now(tm), it would be good if in the future we could encrypt dfs put/get. I'm not so concerned about connectivity between grid nodes (at least right now).

I suspect this JIRA needs to get broken down into multiple subtasks.

[Raghu Angadi]

should it be "hftps:" rather than "hsftp:"?

[Doug Cutting]

> it would be good if in the future we could encrypt dfs put/get [ ... ]

Hftp: uris can be used most places that hdfs: uris can be used, including put/get. The big limitations at present are that hftp does not support:

1. random access – hftp mapreduce inputs cannot be split; and
2. locality hints – hftp mapreduce inputs are not localized.

Currently, to use an hftp filesystem as a mapreduce input, one would have to set mapred.min.split.size=0xffffffffffffffff, but, other than that, it should work.

So I think making the hftp protocol secure will permit the uses you have in mind, no?

[Doug Cutting]

> should it be "hftps:" rather than "hsftp:"?

Whatever. Both SFTP and FTPS refer to secure file-transfer protocols. The former does not require certs, while the latter (like HTTPS) does. So maybe FTPS is a better analogy, but HSFTP reads better to me, as the Hadoop Secure File Transfer Protocol.

[Chris Douglas]

This roughly follows what Doug outlined above. I added a SunJsseListener to the namenode and datanode StatusHttpServer, initialized iff the keystore location is specified. The keystore properties- including passwords- are specified in another resource, specified in the config. I added HsftpFileSystem to handle the client-side and included a redirect to a ssl-capable datanode port from the NameNode servlet, assumed to be static (avoiding the protocol version bump).

[Tsz Wo (Nicholas), SZE]

2239-0.patch: Codes are good. It even make the origin codes better. Below are some comments

• It may not be good to store passwords in the config file. We should, at least having a option, to let the user enter passwords in runtime (during starting namenode, datanodes).

• need descriptions for new config properties

• need javadoc for protected members.

• need unit tests

[Chris Douglas]

This patch adds some documentation, per Nicholas's recommendation. It does not include any test cases, as the requirements for configuring ssl are somewhat onerous and- in my limited experience- not amenable to automation in a test case. Lacking certs, it was tested with Firefox and appears correct. The passwords are stored in a config file, which is regrettable, but the resource storing them need only be on the classpath. Getting this information is out-of-band as it is, and an auxiliary config file seemed the most expedient and mostly-correct option available. For Right Now(tm), it should suffice.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12376886/2239-1.patch
against trunk revision 619744.

@author +1. The patch does not contain any @author tags.

tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

javadoc +1. The javadoc tool did not generate any warning messages.

javac -1. The applied patch generated 616 javac compiler warnings (more than the trunk's current 614 warnings).

release audit -1. The applied patch generated 191 release audit warnings (more than the trunk's current 190 warnings).

findbugs +1. The patch does not introduce any new Findbugs warnings.

core tests +1. The patch passed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1881/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1881/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1881/artifact/trunk/build/test/checkstyle-errors.html
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1881/artifact/trunk/current/releaseAuditDiffWarnings.txt
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1881/console

This message is automatically generated.

[Owen O'Malley]

We need to do the paper work part of putting encryption into Hadoop before this can be committed.

[Chris Douglas]

Replace deprecated SunJsseListener with SslListener

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12377862/2239-2.patch
against trunk revision 619744.

@author +1. The patch does not contain any @author tags.

tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit -1. The applied patch generated 194 release audit warnings (more than the trunk's current 193 warnings).

findbugs +1. The patch does not introduce any new Findbugs warnings.

core tests +1. The patch passed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1969/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1969/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1969/artifact/trunk/build/test/checkstyle-errors.html
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1969/artifact/trunk/current/releaseAuditDiffWarnings.txt
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1969/console

This message is automatically generated.

[Tsz Wo (Nicholas), SZE]

+1 Codes look good.

• The hsftp port is fixed across a cluster. So this feature will not work for running more than one datanodes in the same machine. Moreover, we have to make sure that more than one datanodes can run in the same machine when this feature is disabled.

• We probably should support the random hsftp ports later.

• We need to add more documentation later.

Other minor comments:

• Remove "import org.mortbay.http.JsseListener" in StatusHttpServer.

• The codes for initializing infoServer in FSNamesystem and DataNode are similar. Why not make a static method in StatusHttpServer?

[Chris Douglas]

I just committed this

[Hudson]

Integrated in Hadoop-trunk #433 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/433/)