Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11339

Reuse buffer for Hadoop RPC

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • ipc, performance
    • None

    Description

      For Hadoop RPCs, we will try to reuse the available connections.
      But when we process each rpc in the same connection, we will allocate a fresh heap byte buffer to store the rpc bytes data. The rpc message may be very large, i.e., datanode blocks report.

      There is chance to cause full gc as discussed in HDFS-7435

      Attachments

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-7435 PB encoding of block reports is very inefficient

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              hitliuyi Yi Liu
              hitliuyi Yi Liu
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated: