Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4953

enable HDFS local reads via mmap

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0
    • HDFS-4949
    • None
    • None

    Description

      Currently, the short-circuit local read pathway allows HDFS clients to access files directly without going through the DataNode. However, all of these reads involve a copy at the operating system level, since they rely on the read() / pread() / etc family of kernel interfaces.

      We would like to enable HDFS to read local files via mmap. This would enable truly zero-copy reads.

      In the initial implementation, zero-copy reads will only be performed when checksums were disabled. Later, we can use the DataNode's cache awareness to only perform zero-copy reads when we know that checksum has already been verified.

      Attachments

        1. benchmark.png
          19 kB
          Todd Lipcon
        2. HDFS-4953.001.patch
          56 kB
          Colin McCabe
        3. HDFS-4953.002.patch
          99 kB
          Colin McCabe
        4. HDFS-4953.003.patch
          101 kB
          Colin McCabe
        5. HDFS-4953.004.patch
          102 kB
          Colin McCabe
        6. HDFS-4953.005.patch
          102 kB
          Colin McCabe
        7. HDFS-4953.006.patch
          104 kB
          Colin McCabe
        8. HDFS-4953.007.patch
          114 kB
          Colin McCabe
        9. HDFS-4953.008.patch
          115 kB
          Colin McCabe

        Issue Links

          Activity

            People

              cmccabe Colin McCabe
              cmccabe Colin McCabe
              Votes:
              0 Vote for this issue
              Watchers:
              38 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: