[HDFS-10860] Switch HttpFS from Tomcat to Jetty

• Type: New Feature
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 3.0.0-alpha4
• Component/s: httpfs
• Target Version/s: 3.0.0-alpha4
• Hadoop Flags: Incompatible change, Reviewed

Release Note

<!-- markdown -->

The following environment variables are deprecated. Set the corresponding
configuration properties instead.

Environment Variable | Configuration Property | Configuration File
----------------------------|------------------------------|--------------------
HTTPFS_TEMP | hadoop.http.temp.dir | httpfs-site.xml
HTTPFS_HTTP_PORT | hadoop.httpfs.http.port | httpfs-site.xml
HTTPFS_MAX_HTTP_HEADER_SIZE | hadoop.http.max.request.header.size and hadoop.http.max.response.header.size | httpfs-site.xml
HTTPFS_MAX_THREADS | hadoop.http.max.threads | httpfs-site.xml
HTTPFS_SSL_ENABLED | hadoop.httpfs.ssl.enabled | httpfs-site.xml
HTTPFS_SSL_KEYSTORE_FILE | ssl.server.keystore.location | ssl-server.xml
HTTPFS_SSL_KEYSTORE_PASS | ssl.server.keystore.password | ssl-server.xml

These default HTTP Services have been added.

Name | Description
-------------------|------------------------------------
/conf | Display configuration properties
/jmx | Java JMX management interface
/logLevel | Get or set log level per class
/logs | Display log files
/stacks | Display JVM stacks
/static/index.html | The static home page

Script httpfs.sh has been deprecated, use `hdfs httpfs` instead. The new scripts are based on the Hadoop shell scripting framework. `hadoop daemonlog` is supported. SSL configurations are read from ssl-server.xml.

• Assignee: John Zhuge
• Reporter: John Zhuge
• Created: 13/Sep/16 05:08
• Updated: 01/Sep/17 15:43
• Resolved: 07/Feb/17 18:00

Description

The Tomcat 6 we are using will reach EOL at the end of 2017. While there are other good options, I would propose switching to Jetty 9 for the following reasons:

• Easier migration. Both Tomcat and Jetty are based on Servlet Containers, so we don't have to change client code that much. It would require more work to switch to JAX-RS.
• Well established.
• Good performance and scalability.

Other alternatives:

• Jersey + Grizzly
• Tomcat 8

Your opinions will be greatly appreciated.

Issue Links

• breaks: HDFS-11665 HttpFSServerWebServer$deprecateEnv may leak secret
• breaks: HDFS-12114 Consistent HttpFS property names
• breaks: HDFS-12166 Do not deprecate HTTPFS_TEMP
• depends upon: HADOOP-13992 KMS should load SSL configuration the same way as SSLFactory
• depends upon: HADOOP-13597 Switch KMS from Tomcat to Jetty
• depends upon: HADOOP-10075 Update jetty dependency to version 9
• depends upon: HADOOP-13990 Document KMS usage of CredentialProvider API
• is depended upon by: HADOOP-14367 Remove unused setting from pom.xml
• is depended upon by: HADOOP-14375 Remove tomcat support from hadoop-functions.sh
• is duplicated by: HDFS-11176 httpfs.sh run fails due to missing temp dir
• is duplicated by: HDFS-11202 httpfs.sh will not run when temp dir does not exist
• is related to: HADOOP-14060 HTTP servlet /logs should require authentication and authorization
• relates to: HDFS-11202 httpfs.sh will not run when temp dir does not exist
• supercedes: HADOOP-12809 Move to tomcat 8.0.21 to support different passwords for key and keystore on HTTPFS/KMS using SSL
• links to: Simple KMS and HttpFS tests