Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4953

enable HDFS local reads via mmap

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0
    • HDFS-4949
    • None
    • None

    Description

      Currently, the short-circuit local read pathway allows HDFS clients to access files directly without going through the DataNode. However, all of these reads involve a copy at the operating system level, since they rely on the read() / pread() / etc family of kernel interfaces.

      We would like to enable HDFS to read local files via mmap. This would enable truly zero-copy reads.

      In the initial implementation, zero-copy reads will only be performed when checksums were disabled. Later, we can use the DataNode's cache awareness to only perform zero-copy reads when we know that checksum has already been verified.

      Attachments

        1. benchmark.png
          19 kB
          Todd Lipcon
        2. HDFS-4953.001.patch
          56 kB
          Colin McCabe
        3. HDFS-4953.002.patch
          99 kB
          Colin McCabe
        4. HDFS-4953.003.patch
          101 kB
          Colin McCabe
        5. HDFS-4953.004.patch
          102 kB
          Colin McCabe
        6. HDFS-4953.005.patch
          102 kB
          Colin McCabe
        7. HDFS-4953.006.patch
          104 kB
          Colin McCabe
        8. HDFS-4953.007.patch
          114 kB
          Colin McCabe
        9. HDFS-4953.008.patch
          115 kB
          Colin McCabe

        Issue Links

        blocks

        Improvement - An improvement or enhancement to an existing feature or task. HDFS-5101 ZCR should work with blocks bigger than 2 GB

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Open
        is duplicated by

        New Feature - A new feature of the product, which has yet to be developed. HDFS-3053 Support for true zero-copy (mmap-based) reads

        • Major - Major loss of function.
        • Resolved
        is part of

        New Feature - A new feature of the product, which has yet to be developed. HDFS-5260 Merge zero-copy memory-mapped HDFS client reads to trunk and branch-2.

        • Major - Major loss of function.
        • Closed
        is related to

        Sub-task - The sub-task of the issue HDFS-5191 revisit zero-copy API in FSDataInputStream to make it more intuitive

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. HDFS-5109 Document new zero-copy short-circuit local reads feature

        • Major - Major loss of function.
        • Open
        relates to

        New Feature - A new feature of the product, which has yet to be developed. HDFS-4949 Centralized cache management in HDFS

        • Major - Major loss of function.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. HDFS-6007 Update documentation about short-circuit local reads

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Closed

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            cmccabe Colin McCabe
            cmccabe Colin McCabe
            Votes:
            0 Vote for this issue
            Watchers:
            38 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment