Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4797

RPC Server can leave a lot of direct buffers

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.17.0
    • Fix Version/s: 0.18.3, 0.19.1
    • Component/s: ipc
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Improve how RPC server reads and writes large buffers. Avoids soft-leak of direct buffers and excess copies in NIO layer.

      Description

      RPC server unwittingly can soft-leak direct buffers. One observed case is that one of the namenodes at Yahoo took 40GB of virtual memory though it was configured for 24GB memory. Most of the memory outside Java heap expected to be direct buffers. This shown to be because of how RPC server reads and writes serialized data. The cause and proposed fix are in following comment.

        Attachments

        1. HADOOP-4797.patch
          5 kB
          Raghu Angadi
        2. HADOOP-4797-branch-18.patch
          5 kB
          Raghu Angadi
        3. TestRpcCpu.patch
          6 kB
          Raghu Angadi
        4. TestRpcCpu.patch
          5 kB
          Raghu Angadi
        5. HADOOP-4797.patch
          5 kB
          Raghu Angadi
        6. HADOOP-4797.patch
          5 kB
          Raghu Angadi
        7. HADOOP-4797-branch-18.patch
          5 kB
          Raghu Angadi
        8. HADOOP-4797-branch-18.patch
          5 kB
          Raghu Angadi
        9. HADOOP-4797-branch-18.patch
          4 kB
          Raghu Angadi

          Activity

            People

            • Assignee:
              rangadi Raghu Angadi
              Reporter:
              rangadi Raghu Angadi
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: