[HADOOP-8942] Thundering herd of RPCs with large responses leads to OOM

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.23.3
• Component/s: ipc
• Assignee: Unassigned
• Reporter: Jason Darrell Lowe
• Created: 18/Oct/12 19:28
• Updated: 29/Jan/14 20:12

Description

When a large number of clients are all making calls with large amounts of response data then the IPC server can exhaust memory. See MAPREDUCE-4730 for an example of this.

There does not appear to be any flow control between the server's handler threads and the responder thread. If a handler thread cannot write out all of the response data without blocking, it queues up the remainder for the responder thread and goes back to the next call in the call queue. If there are enough clients, this can cause the handler threads to overwhelm the heap by queueing response data faster than it can be processed.

Issue Links

• is duplicated by: HADOOP-10233 RPC lacks output flow control
• is related to: MAPREDUCE-4730 AM crashes due to OOM while serving up map task completion events
• is related to: HDFS-5788 listLocatedStatus response can be very large