[HADOOP-9194] RPC Support for QoS

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.2-alpha
• Fix Version/s: 2.1.0-beta
• Component/s: ipc
• Target Version/s: 2.1.0-beta
• Hadoop Flags: Incompatible change, Reviewed
• Release Note: Part of the RPC version 9 change. A service class byte is added after the version byte.
• Assignee: Junping Du
• Reporter: Luke Lu
• Created: 10/Jan/13 00:07
• Updated: 28/Sep/15 20:50
• Resolved: 26/Mar/13 23:31

Description

One of the next frontiers of Hadoop performance is QoS (Quality of Service). We need QoS support to fight the inevitable "buffer bloat" (including various queues, which are probably necessary for throughput) in our software stack. This is important for mixed workload with different latency and throughput requirements (e.g. OLTP vs OLAP, batch and even compaction I/O) against the same DFS.

Any potential bottleneck will need to be managed by QoS mechanisms, starting with RPC.

How about adding a one byte DS (differentiated services) field (a la the 6-bit DS field in IP header) in the RPC header to facilitate the QoS mechanisms (in separate JIRAs)? The byte at a fixed offset (how about 0?) of the header is helpful for implementing high performance QoS mechanisms in switches (software or hardware) and servers with minimum decoding effort.

Issue Links

• duplicates: HDFS-945 Make NameNode resilient to DoS attacks (malicious or otherwise)
• is blocked by: HADOOP-9380 Add totalLength to rpc response
• relates to: HDFS-4412 Support HDFS IO throttling
• relates to: HDFS-237 Better handling of dfsadmin command when namenode is slow
• relates to: HADOOP-9640 RPC Congestion Control with FairCallQueue