[HADOOP-9640] RPC Congestion Control with FairCallQueue

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0, 3.0.0-alpha1
• Labels: hdfs qos rpc
• Release Note: Enable optional RPC-level priority to combat congestion and make request latencies more consistent.
• Assignee: Chris Li
• Reporter: Xiaobo Peng
• Created: 11/Jun/13 22:27
• Updated: 02/Oct/19 17:14
• Resolved: 08/Feb/19 17:40

Description

For an easy-to-read summary see: http://www.ebaytechblog.com/2014/08/21/quality-of-service-in-hadoop/

Several production Hadoop cluster incidents occurred where the Namenode was overloaded and failed to respond.

We can improve quality of service for users during namenode peak loads by replacing the FIFO call queue with a Fair Call Queue. (this plan supersedes rpc-congestion-control-draft-plan).

Excerpted from the communication of one incident, “The map task of a user was creating huge number of small files in the user directory. Due to the heavy load on NN, the JT also was unable to communicate with NN...The cluster became responsive only once the job was killed.”

Excerpted from the communication of another incident, “Namenode was overloaded by GetBlockLocation requests (Correction: should be getFileInfo requests. the job had a bug that called getFileInfo for a nonexistent file in an endless loop). All other requests to namenode were also affected by this and hence all jobs slowed down. Cluster almost came to a grinding halt…Eventually killed jobtracker to kill all jobs that are running.”

Excerpted from HDFS-945, “We've seen defective applications cause havoc on the NameNode, for e.g. by doing 100k+ 'listStatus' on very large directories (60k files) etc.”

Issue Links

• depends upon: HADOOP-10508 RefreshCallQueue fails when authorization is enabled
• is related to: HADOOP-9194 RPC Support for QoS
• is related to: HDFS-945 Make NameNode resilient to DoS attacks (malicious or otherwise)
• relates to: HDFS-237 Better handling of dfsadmin command when namenode is slow
• relates to: HADOOP-10598 Support configurable RPC fair share
• relates to: HADOOP-10599 Support prioritization of DN RPCs over client RPCs
• relates to: HADOOP-13029 Have FairCallQueue try all lower priority sub queues before backoff
• relates to: HADOOP-15481 Emit FairCallQueue stats as metrics
• relates to: HADOOP-10286 Allow RPCCallBenchmark to benchmark calls by different users

Sub-Tasks

• Refactor to make CallQueue pluggable: Closed
• Allow CallQueue impls to be swapped at runtime (part 1: internals) Depends on: subtask1: Resolved
• Admin interface to swap callqueue at runtime: Closed
• Make Schedulables return a configurable identity of user or group: Closed
• Create multiplexer, a requirement for the fair queue: Closed
• Create a scheduler, which assigns schedulables a priority level: Closed
• Create a FairCallQueue: a multi-level call queue which schedules incoming calls and multiplexes outgoing calls: Closed
• Add metrics to the HistoryRpcScheduler: Resolved
• Make Scheduler and Multiplexer swappable: Resolved
• RPC Server signals backoff to clients when all request queues are full: Resolved
• Add Documentation for FairCallQueue: Resolved