Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14740

Recover data blocks from persistent memory read cache during datanode restarts

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0, 3.1.4, 3.2.2
    • Component/s: caching, datanode
    • Labels:
      None

      Description

      In HDFS-13762, persistent memory (PM) is enabled in HDFS centralized cache management. Even though PM can persist cache data, for simplifying the initial implementation, the previous cache data will be cleaned up during DataNode restarts. Here, we are proposing to improve HDFS PM cache by taking advantage of PM's data persistence characteristic, i.e., recovering the status for cached data, if any, when DataNode restarts, thus, cache warm up time can be saved for user.

        Attachments

        1. HDFS-14740-branch-3.1-001.patch
          59 kB
          Feilong He
        2. HDFS-14740-branch-3.2-001.patch
          59 kB
          Feilong He
        3. HDFS-14740.009.patch
          59 kB
          Feilong He
        4. HDFS-14740-branch-3.1-000.patch
          59 kB
          Feilong He
        5. HDFS-14740-branch-3.2-000.patch
          59 kB
          Feilong He
        6. HDFS-14740.008.patch
          59 kB
          Feilong He
        7. HDFS-14740.007.patch
          54 kB
          Feilong He
        8. HDFS_Persistent_Read-Cache_Test-v2.pdf
          56 kB
          Feilong He
        9. HDFS_Persistent_Read-Cache_Test-v1.1.pdf
          56 kB
          Rui Mo
        10. HDFS_Persistent_Read-Cache_Test-v1.pdf
          55 kB
          Feilong He
        11. HDFS_Persistent_Read-Cache_Design-v1.pdf
          111 kB
          Feilong He
        12. HDFS-14740.006.patch
          54 kB
          Rui Mo
        13. HDFS-14740.005.patch
          54 kB
          Rui Mo
        14. HDFS-14740.004.patch
          53 kB
          Rui Mo
        15. HDFS-14740.003.patch
          53 kB
          Rui Mo
        16. HDFS-14740.002.patch
          42 kB
          Rui Mo
        17. HDFS-14740.001.patch
          42 kB
          Rui Mo
        18. HDFS-14740.000.patch
          41 kB
          Feilong He

          Issue Links

            Activity

              People

              • Assignee:
                PhiloHe Feilong He
                Reporter:
                PhiloHe Feilong He
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: