[HDFS-7279] Use netty to implement DatanodeWebHdfsMethods

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.7.0
• Component/s: datanode, webhdfs
• Hadoop Flags: Reviewed
• Assignee: Haohui Mai
• Reporter: Haohui Mai
• Created: 23/Oct/14 00:08
• Updated: 18/Feb/19 02:30
• Resolved: 17/Nov/14 20:56

Description

Currently the DN implements all related webhdfs functionality using jetty. As the current jetty version the DN used (jetty 6) lacks of fine-grained buffer and connection management, DN often suffers from long latency and OOM when its webhdfs component is under sustained heavy load.

This jira proposes to implement the webhdfs component in DN using netty, which can be more efficient and allow more finer-grain controls on webhdfs.

Issue Links

• Blocked: HDFS-14290 Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by DatanodeWebHdfsMethods
• Blocked: HDFS-13899 Unexpected message type: PooledUnsafeDirectByteBuf when get datanode info by DatanodeWebHdfsMethods
• breaks: HDFS-7959 WebHdfs logging is missing on Datanode
• breaks: HDFS-13126 Backport [HDFS-7959] to branch-2.7 to re-enable HTTP request logging for WebHDFS
• breaks: HDFS-7816 Unable to open webhdfs paths with "+"
• breaks: HDFS-7818 OffsetParam should return the default value instead of throwing NPE when the value is unspecified
• breaks: HDFS-7945 The WebHdfs system on DN does not honor the length parameter
• breaks: HDFS-8572 DN always uses HTTP/localhost@REALM principals in SPNEGO
• breaks: HDFS-7406 SimpleHttpProxyHandler puts incorrect "Connection: Close" header
• is related to: HDFS-7404 Remove o.a.h.hdfs.server.datanode.web.resources