[KNOX-1524] Hive "select *" performance evaluation

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Information Provided
• Fix Version/s: 1.2.0
• Assignee: Kevin Risden
• Reporter: Kevin Risden
• Created: 15/Oct/18 13:51
• Updated: 28/Mar/19 13:57
• Resolved: 02/Nov/18 20:39

Description

While looking at WebHDFS performance in KNOX-1221, I decided to look a bit more into performance for common use cases. Hive performance is another area that could use some research.

Use "select * ... limit" to get a comparison of raw return speed from HiveServer2. This should show how fast results can be streamed through HiveServer2 and Knox. Compare the results to "hdfs dfs -text" since this will render the data directly from HDFS. This should give comparisons for the difference in overhead between HDFS, HiveServer2 binary, HiveServer2 HTTP, and HiveServer2 HTTP with Knox.

Issue Links

• is related to: KNOX-165 Stress testing
• is related to: KNOX-1530 Improve Gzip Compression Handling Performance
• is related to: KNOX-1525 HBase "scan" performance evaluation
• relates to: HIVE-20621 GetOperationStatus called in resultset.next causing incremental slowness
• relates to: KNOX-1221 WebHDFS read/write performance limitations
• relates to: HIVE-17194 JDBC: Implement Gzip compression for HTTP mode