Hadoop Common
  1. Hadoop Common
  2. HADOOP-4797

RPC Server can leave a lot of direct buffers

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.17.0
    • Fix Version/s: 0.18.3, 0.19.1
    • Component/s: ipc
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Improve how RPC server reads and writes large buffers. Avoids soft-leak of direct buffers and excess copies in NIO layer.

      Description

      RPC server unwittingly can soft-leak direct buffers. One observed case is that one of the namenodes at Yahoo took 40GB of virtual memory though it was configured for 24GB memory. Most of the memory outside Java heap expected to be direct buffers. This shown to be because of how RPC server reads and writes serialized data. The cause and proposed fix are in following comment.

      1. HADOOP-4797.patch
        5 kB
        Raghu Angadi
      2. HADOOP-4797.patch
        5 kB
        Raghu Angadi
      3. HADOOP-4797.patch
        5 kB
        Raghu Angadi
      4. HADOOP-4797-branch-18.patch
        5 kB
        Raghu Angadi
      5. HADOOP-4797-branch-18.patch
        5 kB
        Raghu Angadi
      6. HADOOP-4797-branch-18.patch
        5 kB
        Raghu Angadi
      7. HADOOP-4797-branch-18.patch
        4 kB
        Raghu Angadi
      8. TestRpcCpu.patch
        6 kB
        Raghu Angadi
      9. TestRpcCpu.patch
        5 kB
        Raghu Angadi

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Raghu Angadi
            Reporter:
            Raghu Angadi
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development