Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14740

Recover data blocks from persistent memory read cache during datanode restarts

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.3.0, 3.1.4, 3.2.2
    • caching, datanode
    • None

    Description

      In HDFS-13762, persistent memory (PM) is enabled in HDFS centralized cache management. Even though PM can persist cache data, for simplifying the initial implementation, the previous cache data will be cleaned up during DataNode restarts. Here, we are proposing to improve HDFS PM cache by taking advantage of PM's data persistence characteristic, i.e., recovering the status for cached data, if any, when DataNode restarts, thus, cache warm up time can be saved for user.

      Attachments

        1. HDFS_Persistent_Read-Cache_Design-v1.pdf
          111 kB
          Feilong He
        2. HDFS_Persistent_Read-Cache_Test-v1.1.pdf
          56 kB
          Rui Mo
        3. HDFS_Persistent_Read-Cache_Test-v1.pdf
          55 kB
          Feilong He
        4. HDFS_Persistent_Read-Cache_Test-v2.pdf
          56 kB
          Feilong He
        5. HDFS-14740.000.patch
          41 kB
          Feilong He
        6. HDFS-14740.001.patch
          42 kB
          Rui Mo
        7. HDFS-14740.002.patch
          42 kB
          Rui Mo
        8. HDFS-14740.003.patch
          53 kB
          Rui Mo
        9. HDFS-14740.004.patch
          53 kB
          Rui Mo
        10. HDFS-14740.005.patch
          54 kB
          Rui Mo
        11. HDFS-14740.006.patch
          54 kB
          Rui Mo
        12. HDFS-14740.007.patch
          54 kB
          Feilong He
        13. HDFS-14740.008.patch
          59 kB
          Feilong He
        14. HDFS-14740.009.patch
          59 kB
          Feilong He
        15. HDFS-14740-branch-3.1-000.patch
          59 kB
          Feilong He
        16. HDFS-14740-branch-3.1-001.patch
          59 kB
          Feilong He
        17. HDFS-14740-branch-3.2-000.patch
          59 kB
          Feilong He
        18. HDFS-14740-branch-3.2-001.patch
          59 kB
          Feilong He

        Issue Links

        is related to

        Bug - A problem which impairs or prevents the functions of the product. HDFS-15788 Correct the statement for pmem cache to reflect cache persistence support

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        New Feature - A new feature of the product, which has yet to be developed. HDFS-13762 Support non-volatile storage class memory(SCM) in HDFS cache directives

        • Major - Major loss of function.
        • Resolved
        relates to

        Improvement - An improvement or enhancement to an existing feature or task. HDFS-17012 Remove unused DFSConfigKeys#DFS_DATANODE_PMEM_CACHE_DIRS_DEFAULT

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            PhiloHe Feilong He
            PhiloHe Feilong He
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment