[HDFS-5502] Fix HTTPS support in HsftpFileSystem

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Hadoop Flags: Reviewed

Release Note

Fix the https support in HsftpFileSystem. With the change the client now verifies the server certificate. In particular, client side will verify the Common Name of the certificate using a strategy specified by the configuration property "hadoop.ssl.hostname.verifier".

• Assignee: Haohui Mai
• Reporter: Haohui Mai
• Created: 12/Nov/13 02:51
• Updated: 24/Feb/14 20:57
• Resolved: 16/Nov/13 01:08

Description

The current implementation of HsftpFileSystem suffers from the following issues:

• It initializes the SSLContext incorrectly. It blindly trusts all server certificates which creates a security hole.
• It tries to cancel delegation token through http, not https, which leads to HDFS-5295.
• It overrides the default socket factory for HttpsConnection. Given the fact that it trusts all server-side certificate, it accidentally disables all checks on server certificates for all https connections.

This jira tracks the effort to fix the above issues.

Issue Links

• depends upon: HDFS-5440 Extract the logic of handling delegation tokens in HftpFileSystem to the TokenAspect class
• depends upon: HDFS-5487 Introduce unit test for TokenAspect
• depends upon: HDFS-5506 Use URLConnectionFactory in DelegationTokenFetcher
• duplicates: HDFS-5392 Unify the initialization of the hsftp and swebhdfs clients
• duplicates: HDFS-5295 hsftp throws an exception in the end on secure cluster with https enabled
• supercedes: HDFS-594 Add support for byte-ranges to hsftp