Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: HDFS-4949
    • Component/s: None
    • Labels:
      None

      Description

      Currently, the short-circuit local read pathway allows HDFS clients to access files directly without going through the DataNode. However, all of these reads involve a copy at the operating system level, since they rely on the read() / pread() / etc family of kernel interfaces.

      We would like to enable HDFS to read local files via mmap. This would enable truly zero-copy reads.

      In the initial implementation, zero-copy reads will only be performed when checksums were disabled. Later, we can use the DataNode's cache awareness to only perform zero-copy reads when we know that checksum has already been verified.

      1. HDFS-4953.001.patch
        56 kB
        Colin Patrick McCabe
      2. HDFS-4953.002.patch
        99 kB
        Colin Patrick McCabe
      3. benchmark.png
        19 kB
        Todd Lipcon
      4. HDFS-4953.003.patch
        101 kB
        Colin Patrick McCabe
      5. HDFS-4953.004.patch
        102 kB
        Colin Patrick McCabe
      6. HDFS-4953.005.patch
        102 kB
        Colin Patrick McCabe
      7. HDFS-4953.006.patch
        104 kB
        Colin Patrick McCabe
      8. HDFS-4953.007.patch
        114 kB
        Colin Patrick McCabe
      9. HDFS-4953.008.patch
        115 kB
        Colin Patrick McCabe

        Issue Links

          Activity

          Colin Patrick McCabe created issue -
          Colin Patrick McCabe made changes -
          Field Original Value New Value
          Attachment 0001-HDFS-4953.patch [ 12592045 ]
          Colin Patrick McCabe made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Colin Patrick McCabe made changes -
          Attachment 0001-HDFS-4953.patch [ 12592045 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-4953.001.patch [ 12592080 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-4953.002.patch [ 12592112 ]
          Colin Patrick McCabe made changes -
          Link This issue relates to HDFS-4949 [ HDFS-4949 ]
          Todd Lipcon made changes -
          Attachment benchmark.png [ 12592116 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-4953.003.patch [ 12592416 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-4953.004.patch [ 12592598 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-4953.005.patch [ 12592633 ]
          Arun C Murthy made changes -
          Affects Version/s 2.3.0 [ 12324588 ]
          Affects Version/s 2.2.0 [ 12324630 ]
          Colin Patrick McCabe made changes -
          Link This issue is duplicated by HDFS-3053 [ HDFS-3053 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-4953.006.patch [ 12594739 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-4953.007.patch [ 12598146 ]
          Colin Patrick McCabe made changes -
          Link This issue blocks HDFS-5101 [ HDFS-5101 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-4953.008.patch [ 12598514 ]
          Andrew Wang made changes -
          Link This issue is related to HDFS-5109 [ HDFS-5109 ]
          Andrew Wang made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Target Version/s
          Fix Version/s HDFS-4949 [ 12324878 ]
          Resolution Fixed [ 1 ]
          Colin Patrick McCabe made changes -
          Link This issue is related to HDFS-5191 [ HDFS-5191 ]
          Chris Nauroth made changes -
          Link This issue is related to HDFS-5260 [ HDFS-5260 ]
          Chris Nauroth made changes -
          Link This issue is part of HDFS-5260 [ HDFS-5260 ]
          Chris Nauroth made changes -
          Link This issue is related to HDFS-5260 [ HDFS-5260 ]
          Arun C Murthy made changes -
          Affects Version/s 2.3.0 [ 12325255 ]
          Affects Version/s 2.4.0 [ 12324588 ]
          Masatake Iwasaki made changes -
          Link This issue relates to HDFS-6007 [ HDFS-6007 ]

            People

            • Assignee:
              Colin Patrick McCabe
              Reporter:
              Colin Patrick McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              37 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development