[KNOX-1530] Improve Gzip Compression Handling Performance

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Assignee: Kevin Risden
• Reporter: Kevin Risden
• Created: 19/Oct/18 15:22
• Updated: 28/Mar/19 13:57
• Resolved: 30/Oct/18 20:02

Description

While looking at KNOX-1524, I found that requesting compressed results can cause performance impacts. Knox currently does the following:

• Apache HttpClient transparently decompresses each request
  • Apache HttpClient 4.1 added support for this - HTTPCLIENT-834

This lead to recompressing some streams (KNOX-732, KNOX-855, KNOX-856) based on MimeTypes. Even if we disableContentCompression, KNOX-565 added the following which should only come into play with the above HttpClient transparent decompression disabled (or multipart Gzip files - KNOX-1518):

• Try to decompress the stream
  • Currently uses try/catch
• Run any rewrite filter rules
• If decompressed, recompress the stream

For many use cases, there is no reason to decompress and recompress the same stream. This is because there are no rewrite rules that apply. One example of this is Hive where beeline requests compression and HiveServer2 added support for returning compressed results with HIVE-17194. Another is with WebHDFS where we don't want to change the content going back to the client.

I am planning to address this in a few pieces:

• Determine if any rewrite rules apply before decompressing
  • If rewrite rules apply, then decompress and recompress as before
  • If rewrite rules do not apply, then copy stream as is
• Remove gzip filter added by KNOX-732
  • Figure out if there is another code path where decompress/recompress should happen
  • We should not have to rely on Jetty to recompress content
• Disable httpclient content compression
  • Need to make sure we handle decompress/recompress where necessary

With all 3 improvements in place we should end up with:

• One place where gzip decompress/recompress happens
• Only decompress/recompress if rewrite rules match
• Performance increases due to skipping unnecessary decompress/recompress

Issue Links

• is related to: KNOX-165 Stress testing
• relates to: KNOX-732 Knox does not recompress javascript resources
• relates to: KNOX-855 Add "application/x-javascript" mime type to the list of compressed resources
• relates to: KNOX-1221 WebHDFS read/write performance limitations
• relates to: KNOX-856 Document gzip compression options
• relates to: HTTPCLIENT-834 Transparent Content Coding support
• relates to: KNOX-565 Supporting All the Quick Links on Ambari Dashboard to Go Through Knox
• relates to: HIVE-17194 JDBC: Implement Gzip compression for HTTP mode
• relates to: KNOX-1518 Large HDFS file downloads are incomplete when content is gzipped
• relates to: KNOX-1524 Hive "select *" performance evaluation
• relates to: KNOX-1525 HBase "scan" performance evaluation

Sub-Tasks

• Avoid try/catch checking for gzip stream: Closed
• Only decompress/recompress when there are rewrite rules: Closed
• Disable httpclient transparent content decompression: Closed
• Remove custom gzip helper stream - use commons-compress: Closed
• Remove gzip handler added by KNOX-732: Closed