Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14740

Recover data blocks from persistent memory read cache during datanode restarts

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.3.0, 3.1.4, 3.2.2
    • caching, datanode
    • None

    Description

      In HDFS-13762, persistent memory (PM) is enabled in HDFS centralized cache management. Even though PM can persist cache data, for simplifying the initial implementation, the previous cache data will be cleaned up during DataNode restarts. Here, we are proposing to improve HDFS PM cache by taking advantage of PM's data persistence characteristic, i.e., recovering the status for cached data, if any, when DataNode restarts, thus, cache warm up time can be saved for user.

      Attachments

        1. HDFS-14740.000.patch
          41 kB
          Feilong He
        2. HDFS-14740.001.patch
          42 kB
          Rui Mo
        3. HDFS-14740.002.patch
          42 kB
          Rui Mo
        4. HDFS-14740.003.patch
          53 kB
          Rui Mo
        5. HDFS-14740.004.patch
          53 kB
          Rui Mo
        6. HDFS-14740.005.patch
          54 kB
          Rui Mo
        7. HDFS-14740.006.patch
          54 kB
          Rui Mo
        8. HDFS_Persistent_Read-Cache_Design-v1.pdf
          111 kB
          Feilong He
        9. HDFS_Persistent_Read-Cache_Test-v1.pdf
          55 kB
          Feilong He
        10. HDFS_Persistent_Read-Cache_Test-v1.1.pdf
          56 kB
          Rui Mo
        11. HDFS_Persistent_Read-Cache_Test-v2.pdf
          56 kB
          Feilong He
        12. HDFS-14740.007.patch
          54 kB
          Feilong He
        13. HDFS-14740.008.patch
          59 kB
          Feilong He
        14. HDFS-14740-branch-3.2-000.patch
          59 kB
          Feilong He
        15. HDFS-14740-branch-3.1-000.patch
          59 kB
          Feilong He
        16. HDFS-14740.009.patch
          59 kB
          Feilong He
        17. HDFS-14740-branch-3.2-001.patch
          59 kB
          Feilong He
        18. HDFS-14740-branch-3.1-001.patch
          59 kB
          Feilong He

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-15788 Correct the statement for pmem cache to reflect cache persistence support

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HDFS-13762 Support non-volatile storage class memory(SCM) in HDFS cache directives

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-17012 Remove unused DFSConfigKeys#DFS_DATANODE_PMEM_CACHE_DIRS_DEFAULT

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              PhiloHe Feilong He
              PhiloHe Feilong He
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: