Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4949

Centralized cache management in HDFS

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.3.0, 3.0.0-alpha1
    • 2.3.0
    • datanode, namenode
    • None

    Description

      HDFS currently has no support for managing or exposing in-memory caches at datanodes. This makes it harder for higher level application frameworks like Hive, Pig, and Impala to effectively use cluster memory, because they cannot explicitly cache important datasets or place their tasks for memory locality.

      Attachments

        1. caching-design-doc-2013-07-02.pdf
          270 kB
          Andrew Wang
        2. caching-design-doc-2013-08-09.pdf
          305 kB
          Andrew Wang
        3. caching-design-doc-2013-10-24.pdf
          312 kB
          Colin McCabe
        4. caching-testplan.pdf
          99 kB
          Stephen Chu
        5. hdfs-4949-branch-2.patch
          698 kB
          Andrew Wang
        6. HDFS-4949-consolidated.patch
          503 kB
          Andrew Wang

        Issue Links

        1.
        Add JNI mlock support Sub-task Resolved Andrew Wang Actions
        2.
        Propagate cache status information from the DataNode to the NameNode Sub-task Resolved Andrew Wang Actions
        3.
        Add DataNode support for mlock and munlock Sub-task Resolved Andrew Wang Actions
        4.
        Add cacheRequest/uncacheRequest support to NameNode Sub-task Resolved Colin McCabe Actions
        5.
        NameNode should invoke DataNode APIs to coordinate caching Sub-task Resolved Andrew Wang Actions
        6.
        add RPCs for creating and manipulating cache pools Sub-task Resolved Colin McCabe Actions
        7.
        add command-line support for manipulating cache pools Sub-task Resolved Colin McCabe Actions
        8.
        Add cache status information to datanode heartbeat Sub-task Resolved Andrew Wang Actions
        9.
        add command-line support for manipulating cache directives Sub-task Resolved Colin McCabe Actions
        10.
        miscellaneous cache pool RPC fixes Sub-task Resolved Colin McCabe Actions
        11.
        prettier dfsadmin -listCachePools output Sub-task Resolved Colin McCabe Actions
        12.
        revisit zero-copy API in FSDataInputStream to make it more intuitive Sub-task Resolved Colin McCabe Actions
        13.
        Automatically cache new data added to a cached path Sub-task Resolved Colin McCabe Actions
        14.
        Persist CacheManager state in the edit log Sub-task Resolved Andrew Wang Actions
        15.
        Support for federated cache pools Sub-task Resolved Andrew Wang Actions
        16.
        caching PB cleanups Sub-task Resolved Colin McCabe Actions
        17.
        Move cache pool related CLI commands to CacheAdmin Sub-task Resolved Andrew Wang Actions
        18.
        NameNodeRpcServer must not send back DNA_FINALIZE in reply to a cache report Sub-task Resolved Colin McCabe Actions
        19.
        NativeIO: consolidate getrlimit into NativeIO#getMemlockLimit Sub-task Resolved Colin McCabe Actions
        20.
        Fix some failing unit tests on HDFS-4949 branch Sub-task Resolved Andrew Wang Actions
        21.
        separate PathBasedCacheEntry and PathBasedCacheDirectiveWithId Sub-task Resolved Colin McCabe Actions
        22.
        Refactor PathBasedCache* methods to use a Path rather than a String Sub-task Resolved Chris Nauroth Actions
        23.
        Change PathBasedCacheDirective APIs to be a single value rather than batch Sub-task Resolved Andrew Wang Actions
        24.
        Add requesting user's name to PathBasedCacheEntry Sub-task Resolved Andrew Wang Actions
        25.
        Expose if a block replica is cached in getFileBlockLocations Sub-task Resolved Andrew Wang Actions
        26.
        Fix failing caching unit tests Sub-task Resolved Andrew Wang Actions
        27.
        Do not expose CachePool type in AddCachePoolOp Sub-task Resolved Colin McCabe Actions
        28.
        Add datanode caching metrics Sub-task Closed Andrew Wang Actions
        29.
        add modifyDirective to cacheAdmin Sub-task Resolved Colin McCabe Actions
        30.
        Fix error message when dfs.datanode.max.locked.memory is improperly configured Sub-task Resolved Colin McCabe Actions
        31.
        DNA_CACHE and DNA_UNCACHE should be by blockId only Sub-task Resolved Colin McCabe Actions
        32.
        Add replication field to PathBasedCacheDirective Sub-task Resolved Colin McCabe Actions
        33.
        Allow LightWeightGSet#Iterator to remove elements Sub-task Resolved Colin McCabe Actions
        34.
        recaching improvements Sub-task Closed Colin McCabe Actions
        35.
        In CacheReport, don't send genstamp and length on the wire Sub-task Resolved Colin McCabe Actions
        36.
        fix broken caching unit tests Sub-task Resolved Andrew Wang Actions
        37.
        Loading fsimage fails to find cache pools during namenode startup. Sub-task Resolved Chris Nauroth Actions
        38.
        Concurrent clients that add a cache directive on the same path may prematurely uncache from each other. Sub-task Resolved Chris Nauroth Actions
        39.
        fix race conditions in DN caching and uncaching Sub-task Closed Colin McCabe Actions
        40.
        Add feature documentation for datanode caching. Sub-task Closed Colin McCabe Actions
        41.
        Caching RPCs are AtMostOnce, but do not persist client ID and call ID to edit log. Sub-task Resolved Chris Nauroth Actions
        42.
        Resolve regressions in Windows compatibility on HDFS-4949 branch. Sub-task Resolved Chris Nauroth Actions
        43.
        Fix possible RetryCache hang for caching RPC handlers in FSNamesystem Sub-task Resolved Andrew Wang Actions
        44.
        Fixup test-patch.sh warnings on HDFS-4949 branch Sub-task Resolved Andrew Wang Actions
        45.
        Support TTL on CacheDirectives Sub-task Closed Andrew Wang Actions
        46.
        support cachepool-based limit management in path-based caching Sub-task Closed Andrew Wang Actions
        47.
        better API for getting the cached blocks locations Sub-task Closed Andrew Wang Actions
        48.
        Add byte and file statistics to PathBasedCacheEntry Sub-task Closed Colin McCabe Actions
        49.
        Consistent naming of user-visible caching classes and methods Sub-task Closed Colin McCabe Actions
        50.
        add command-line support for modifyDirective Sub-task Resolved Colin McCabe Actions
        51.
        Consider maximum DN memory, stale status when scheduling recaching Sub-task Resolved Colin McCabe Actions
        52.
        TestPathBasedCacheRequests#testReplicationFactor is flaky Sub-task Closed Andrew Wang Actions
        53.
        improve CacheManipulator interface to allow better unit testing Sub-task Closed Colin McCabe Actions
        54.
        loading cache path directives from edit log doesn't update nextEntryId Sub-task Closed Colin McCabe Actions
        55.
        skip checksums when reading a cached block via non-local reads Sub-task Resolved Colin McCabe Actions
        56.
        fix narrow race condition in TestPathBasedCacheRequests Sub-task Closed Colin McCabe Actions
        57.
        Rename "path.based" caching configuration options Sub-task Resolved Andrew Wang Actions
        58.
        add some more NameNode cache statistics, cache pool stats Sub-task Closed Colin McCabe Actions
        59.
        Refactor tests in TestCacheDirectives Sub-task Resolved Andrew Wang Actions
        60.
        CacheAdmin help should match against non-dashed commands Sub-task Closed Andrew Wang Actions
        61.
        Namenode loops caching and uncaching when data should be uncached Sub-task Closed Andrew Wang Actions
        62.
        Hook up cache directive and pool usage statistics Sub-task Closed Andrew Wang Actions
        63.
        allow BlockReaderLocal to switch between checksumming and not Sub-task Closed Colin McCabe Actions
        64.
        Enforce a max TTL per cache pool Sub-task Closed Andrew Wang Actions
        65.
        Remove dfs.namenode.caching.enabled and improve CRM locking Sub-task Closed Colin McCabe Actions
        66.
        The CacheManager throws a NPE in the DataNode logs when processing cache reports that refer to a block not known to the BlockManager Sub-task Closed Colin McCabe Actions
        67.
        Add per-cache-pool default replication num configuration Sub-task Resolved xupeng Actions

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            andrew.wang Andrew Wang
            andrew.wang Andrew Wang
            Votes:
            0 Vote for this issue
            Watchers:
            102 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment