[HADOOP-4797] RPC Server can leave a lot of direct buffers

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.17.0
• Fix Version/s: 0.18.3, 0.19.1
• Component/s: ipc
• Hadoop Flags: Reviewed
• Release Note: Improve how RPC server reads and writes large buffers. Avoids soft-leak of direct buffers and excess copies in NIO layer.
• Assignee: Raghu Angadi
• Reporter: Raghu Angadi
• Created: 06/Dec/08 02:22
• Updated: 30/Jan/09 20:14
• Resolved: 17/Dec/08 00:12

Description

RPC server unwittingly can soft-leak direct buffers. One observed case is that one of the namenodes at Yahoo took 40GB of virtual memory though it was configured for 24GB memory. Most of the memory outside Java heap expected to be direct buffers. This shown to be because of how RPC server reads and writes serialized data. The cause and proposed fix are in following comment.