Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: HDFS-4949
    • Component/s: None
    • Labels:
      None

      Description

      Currently, the short-circuit local read pathway allows HDFS clients to access files directly without going through the DataNode. However, all of these reads involve a copy at the operating system level, since they rely on the read() / pread() / etc family of kernel interfaces.

      We would like to enable HDFS to read local files via mmap. This would enable truly zero-copy reads.

      In the initial implementation, zero-copy reads will only be performed when checksums were disabled. Later, we can use the DataNode's cache awareness to only perform zero-copy reads when we know that checksum has already been verified.

        Attachments

        1. HDFS-4953.008.patch
          115 kB
          Colin P. McCabe
        2. HDFS-4953.007.patch
          114 kB
          Colin P. McCabe
        3. HDFS-4953.006.patch
          104 kB
          Colin P. McCabe
        4. HDFS-4953.005.patch
          102 kB
          Colin P. McCabe
        5. HDFS-4953.004.patch
          102 kB
          Colin P. McCabe
        6. HDFS-4953.003.patch
          101 kB
          Colin P. McCabe
        7. HDFS-4953.002.patch
          99 kB
          Colin P. McCabe
        8. HDFS-4953.001.patch
          56 kB
          Colin P. McCabe
        9. benchmark.png
          19 kB
          Todd Lipcon

          Issue Links

            Activity

              People

              • Assignee:
                cmccabe Colin P. McCabe
                Reporter:
                cmccabe Colin P. McCabe
              • Votes:
                0 Vote for this issue
                Watchers:
                37 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: