Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14740

Recover data blocks from persistent memory read cache during datanode restarts

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.3.0, 3.1.4, 3.2.2
    • caching, datanode
    • None

    Description

      In HDFS-13762, persistent memory (PM) is enabled in HDFS centralized cache management. Even though PM can persist cache data, for simplifying the initial implementation, the previous cache data will be cleaned up during DataNode restarts. Here, we are proposing to improve HDFS PM cache by taking advantage of PM's data persistence characteristic, i.e., recovering the status for cached data, if any, when DataNode restarts, thus, cache warm up time can be saved for user.

      Attachments

        1. HDFS_Persistent_Read-Cache_Design-v1.pdf
          111 kB
          Feilong He
        2. HDFS_Persistent_Read-Cache_Test-v1.1.pdf
          56 kB
          Rui Mo
        3. HDFS_Persistent_Read-Cache_Test-v1.pdf
          55 kB
          Feilong He
        4. HDFS_Persistent_Read-Cache_Test-v2.pdf
          56 kB
          Feilong He
        5. HDFS-14740.000.patch
          41 kB
          Feilong He
        6. HDFS-14740.001.patch
          42 kB
          Rui Mo
        7. HDFS-14740.002.patch
          42 kB
          Rui Mo
        8. HDFS-14740.003.patch
          53 kB
          Rui Mo
        9. HDFS-14740.004.patch
          53 kB
          Rui Mo
        10. HDFS-14740.005.patch
          54 kB
          Rui Mo
        11. HDFS-14740.006.patch
          54 kB
          Rui Mo
        12. HDFS-14740.007.patch
          54 kB
          Feilong He
        13. HDFS-14740.008.patch
          59 kB
          Feilong He
        14. HDFS-14740.009.patch
          59 kB
          Feilong He
        15. HDFS-14740-branch-3.1-000.patch
          59 kB
          Feilong He
        16. HDFS-14740-branch-3.1-001.patch
          59 kB
          Feilong He
        17. HDFS-14740-branch-3.2-000.patch
          59 kB
          Feilong He
        18. HDFS-14740-branch-3.2-001.patch
          59 kB
          Feilong He

        Issue Links

          Activity

            People

              PhiloHe Feilong He
              PhiloHe Feilong He
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: