[KNOX-1221] WebHDFS read/write performance limitations

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Information Provided
• Affects Version/s: 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, 0.14.0, 1.0.0, 1.1.0
• Fix Version/s: 1.2.0
• Component/s: Server

Environment

Knox 1.0.0
RHEL 6.9
JDK 1.8.0_60
Launch string:
/usr/bin/java -Djava.library.path=/opt/knox-1.0.0/ext/native -Xms2g -Xmx8g -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9000 -jar /opt/knox-1.0.0/bin/gateway.jar

• Assignee: Kevin Risden
• Reporter: Georgy
• Created: 21/Mar/18 13:59
• Updated: 28/Mar/19 13:57
• Resolved: 15/Oct/18 12:53

Description

If I use direct connection to WebHDFS from one node I have speed nearly several gigabites/sec when download or upload large files. But if I use knox I have ulpload/download speed only 100mbit/sec from the same node. Found that knox limits speed for one https session. As a workaround I upload large files through knox in parallel but it is not a good way.

Couldn't find any configuration params that can control such behaviour. Is it a code limitation?

Issue Links

• is related to: KNOX-165 Stress testing
• is related to: KNOX-1530 Improve Gzip Compression Handling Performance
• is related to: KNOX-1524 Hive "select *" performance evaluation
• is related to: KNOX-1525 HBase "scan" performance evaluation
• relates to: KNOX-1521 Implement UrlRewriteRequestStream methods for performance