Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13762 Support non-volatile storage class memory(SCM) in HDFS cache directives
  3. HDFS-14402

Use FileChannel.transferTo() method for transferring block to SCM cache

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0
    • Component/s: caching, datanode
    • Labels:

      Description

      We will consider to use transferTo API to improve SCM's cach performace.

      Discussion: 5th comment in HDFS-14355

      Comment: FileMappableBlockLoader : Currently you are reading data from input stream and verifying checksum and writing that buffer to MBB. One thought here is: how about using FileChannel#transferTo API for transferring data from one channel to other natively. and then do mmap on destination file(assuming mmap may be faster in target file) and do checksum verification on it? 

        Attachments

        1. HDFS-14402.000.patch
          13 kB
          Feilong He
        2. HDFS-14402.001.patch
          13 kB
          Feilong He
        3. HDFS-14402.002.patch
          14 kB
          Feilong He
        4. With-Cache-Improvement-Patch.png
          65 kB
          Feilong He
        5. Without-Cache-Improvement-Patch.png
          73 kB
          Feilong He

          Activity

            People

            • Assignee:
              PhiloHe Feilong He
              Reporter:
              PhiloHe Feilong He
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: