Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4953

enable HDFS local reads via mmap

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0
    • HDFS-4949
    • None
    • None

    Description

      Currently, the short-circuit local read pathway allows HDFS clients to access files directly without going through the DataNode. However, all of these reads involve a copy at the operating system level, since they rely on the read() / pread() / etc family of kernel interfaces.

      We would like to enable HDFS to read local files via mmap. This would enable truly zero-copy reads.

      In the initial implementation, zero-copy reads will only be performed when checksums were disabled. Later, we can use the DataNode's cache awareness to only perform zero-copy reads when we know that checksum has already been verified.

      Attachments

        1. benchmark.png
          19 kB
          Todd Lipcon
        2. HDFS-4953.001.patch
          56 kB
          Colin McCabe
        3. HDFS-4953.002.patch
          99 kB
          Colin McCabe
        4. HDFS-4953.003.patch
          101 kB
          Colin McCabe
        5. HDFS-4953.004.patch
          102 kB
          Colin McCabe
        6. HDFS-4953.005.patch
          102 kB
          Colin McCabe
        7. HDFS-4953.006.patch
          104 kB
          Colin McCabe
        8. HDFS-4953.007.patch
          114 kB
          Colin McCabe
        9. HDFS-4953.008.patch
          115 kB
          Colin McCabe

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            cmccabe Colin McCabe
            cmccabe Colin McCabe
            Votes:
            0 Vote for this issue
            Watchers:
            39 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment