[HADOOP-14444] New implementation of ftp and sftp filesystems

• Type: New Feature
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.8.0
• Component/s: fs
• Target Version/s: 3.5.0

Release Note

See README.md file supplied.
Default file systems for ftp/sftp schemas can probably stay and user can choose to use new implementation by specifying -Dfs.<schema>.impl flag

• Assignee: Lukas Waldmann
• Reporter: Lukas Waldmann
• Created: 22/May/17 10:11
• Updated: 04/Jan/24 12:07

Description

Current implementation of FTP and SFTP filesystems have severe limitations and performance issues when dealing with high number of files. Mine patch solve those issues and integrate both filesystems such a way that most of the core functionality is common for both and therefore simplifying the maintainability.

The core features:

• Support for HTTP/SOCKS proxies
• Support for passive FTP
• Support for explicit FTPS (SSL/TLS)
• Support of connection pooling - new connection is not created for every single command but reused from the pool.
  For huge number of files it shows order of magnitude performance improvement over not pooled connections.
• Caching of directory trees. For ftp you always need to list whole directory whenever you ask information about particular file.
  Again for huge number of files it shows order of magnitude performance improvement over not cached connections.
• Support of keep alive (NOOP) messages to avoid connection drops
• Support for Unix style or regexp wildcard glob - useful for listing a particular files across whole directory tree
• Support for reestablishing broken ftp data transfers - can happen surprisingly often
• Support for sftp private keys (including pass phrase)
• Support for keeping passwords, private keys and pass phrase in the jceks key stores

Issue Links

• is related to: HADOOP-5732 Add SFTP FileSystem
• relates to: HADOOP-13759 Split SFTP FileSystem into its own artifact