Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-380

support for persistent connections to improve random read performance.

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Linux 2.6.9-55 , Dual Core Opteron 280 2.4Ghz , 4GB memory

      Description

      preads() establish new connections per request. yourkit java profiles show that this connection overhead is pretty significant on the DataNode.
      I wrote a simple microbenchmark program which does many iterations of pread() from different offsets of a large file. I hacked DFSClient/DataNode code to re-use the same connection/DataNode request handler thread. The performance improvement was 7% when the data is served from disk and 80% when the data is served from the OS page cache.

        Attachments

        1. pread_test.java
          1 kB
          George Wu

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                georgewu George Wu
              • Votes:
                0 Vote for this issue
                Watchers:
                16 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: