Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: HDFS-4949
    • Component/s: None
    • Labels:
      None

      Description

      Currently, the short-circuit local read pathway allows HDFS clients to access files directly without going through the DataNode. However, all of these reads involve a copy at the operating system level, since they rely on the read() / pread() / etc family of kernel interfaces.

      We would like to enable HDFS to read local files via mmap. This would enable truly zero-copy reads.

      In the initial implementation, zero-copy reads will only be performed when checksums were disabled. Later, we can use the DataNode's cache awareness to only perform zero-copy reads when we know that checksum has already been verified.

      1. HDFS-4953.001.patch
        56 kB
        Colin Patrick McCabe
      2. HDFS-4953.002.patch
        99 kB
        Colin Patrick McCabe
      3. benchmark.png
        19 kB
        Todd Lipcon
      4. HDFS-4953.003.patch
        101 kB
        Colin Patrick McCabe
      5. HDFS-4953.004.patch
        102 kB
        Colin Patrick McCabe
      6. HDFS-4953.005.patch
        102 kB
        Colin Patrick McCabe
      7. HDFS-4953.006.patch
        104 kB
        Colin Patrick McCabe
      8. HDFS-4953.007.patch
        114 kB
        Colin Patrick McCabe
      9. HDFS-4953.008.patch
        115 kB
        Colin Patrick McCabe

        Issue Links

          Activity

            People

            • Assignee:
              Colin Patrick McCabe
              Reporter:
              Colin Patrick McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              37 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development