Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4949

Centralized cache management in HDFS

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.3.0, 3.0.0-alpha1
    • 2.3.0
    • datanode, namenode
    • None

    Description

      HDFS currently has no support for managing or exposing in-memory caches at datanodes. This makes it harder for higher level application frameworks like Hive, Pig, and Impala to effectively use cluster memory, because they cannot explicitly cache important datasets or place their tasks for memory locality.

      Attachments

        1. caching-design-doc-2013-07-02.pdf
          270 kB
          Andrew Wang
        2. caching-design-doc-2013-08-09.pdf
          305 kB
          Andrew Wang
        3. caching-design-doc-2013-10-24.pdf
          312 kB
          Colin McCabe
        4. caching-testplan.pdf
          99 kB
          Stephen Chu
        5. hdfs-4949-branch-2.patch
          698 kB
          Andrew Wang
        6. HDFS-4949-consolidated.patch
          503 kB
          Andrew Wang

        Issue Links

          is related to

          Sub-task - The sub-task of the issue HDFS-9957 HDFS's use of mlock() is not portable

          • Major - Major loss of function.
          • Open

          Sub-task - The sub-task of the issue HDFS-5203 Concurrent clients that add a cache directive on the same path may prematurely uncache from each other.

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HDFS-5385 Caching RPCs are AtMostOnce, but do not persist client ID and call ID to edit log.

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HDFS-5388 Loading fsimage fails to find cache pools during namenode startup.

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-5195 Prevent passing null pointer to mlock and munlock.

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-5266 ElasticByteBufferPool#Key does not implement equals.

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-5269 Attempting to remove a cache directive fails with NullPointerException.

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-5313 NameNode hangs during startup trying to apply OP_ADD_PATH_BASED_CACHE_DIRECTIVE.

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-5373 hdfs cacheadmin -addDirective short usage does not mention -replication parameter.

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. YARN-1488 Allow containers to delegate resources to another container

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. HDFS-4953 enable HDFS local reads via mmap

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-5197 Document dfs.cachereport.intervalMsec in hdfs-default.xml.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-6347 ZeroCopy read path for ORC RecordReader

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HDFS-2832 Enable support for heterogeneous storages in HDFS - DN as a collection of storages

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-16650 Optimize the cost of obtaining timestamps in Centralized cache management

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • In Progress

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-5202 Support Centralized Cache Management on Windows.

          • Major - Major loss of function.
          • Closed
          1.
          		 Add JNI mlock support Sub-task Resolved Andrew Wang
          2.
          		 Propagate cache status information from the DataNode to the NameNode Sub-task Resolved Andrew Wang
          3.
          		 Add DataNode support for mlock and munlock Sub-task Resolved Andrew Wang
          4.
          		 Add cacheRequest/uncacheRequest support to NameNode Sub-task Resolved Colin McCabe
          5.
          		 NameNode should invoke DataNode APIs to coordinate caching Sub-task Resolved Andrew Wang
          6.
          		 add RPCs for creating and manipulating cache pools Sub-task Resolved Colin McCabe
          7.
          		 add command-line support for manipulating cache pools Sub-task Resolved Colin McCabe
          8.
          		 Add cache status information to datanode heartbeat Sub-task Resolved Andrew Wang
          9.
          		 add command-line support for manipulating cache directives Sub-task Resolved Colin McCabe
          10.
          		 miscellaneous cache pool RPC fixes Sub-task Resolved Colin McCabe
          11.
          		 prettier dfsadmin -listCachePools output Sub-task Resolved Colin McCabe
          12.
          		 revisit zero-copy API in FSDataInputStream to make it more intuitive Sub-task Resolved Colin McCabe
          13.
          		 Automatically cache new data added to a cached path Sub-task Resolved Colin McCabe
          14.
          		 Persist CacheManager state in the edit log Sub-task Resolved Andrew Wang
          15.
          		 Support for federated cache pools Sub-task Resolved Andrew Wang
          16.
          		 caching PB cleanups Sub-task Resolved Colin McCabe
          17.
          		 Move cache pool related CLI commands to CacheAdmin Sub-task Resolved Andrew Wang
          18.
          		 NameNodeRpcServer must not send back DNA_FINALIZE in reply to a cache report Sub-task Resolved Colin McCabe
          19.
          		 NativeIO: consolidate getrlimit into NativeIO#getMemlockLimit Sub-task Resolved Colin McCabe
          20.
          		 Fix some failing unit tests on HDFS-4949 branch Sub-task Resolved Andrew Wang
          21.
          		 separate PathBasedCacheEntry and PathBasedCacheDirectiveWithId Sub-task Resolved Colin McCabe
          22.
          		 Refactor PathBasedCache* methods to use a Path rather than a String Sub-task Resolved Chris Nauroth
          23.
          		 Change PathBasedCacheDirective APIs to be a single value rather than batch Sub-task Resolved Andrew Wang
          24.
          		 Add requesting user's name to PathBasedCacheEntry Sub-task Resolved Andrew Wang
          25.
          		 Expose if a block replica is cached in getFileBlockLocations Sub-task Resolved Andrew Wang
          26.
          		 Fix failing caching unit tests Sub-task Resolved Andrew Wang
          27.
          		 Do not expose CachePool type in AddCachePoolOp Sub-task Resolved Colin McCabe
          28.
          		 Add datanode caching metrics Sub-task Closed Andrew Wang
          29.
          		 add modifyDirective to cacheAdmin Sub-task Resolved Colin McCabe
          30.
          		 Fix error message when dfs.datanode.max.locked.memory is improperly configured Sub-task Resolved Colin McCabe
          31.
          		 DNA_CACHE and DNA_UNCACHE should be by blockId only Sub-task Resolved Colin McCabe
          32.
          		 Add replication field to PathBasedCacheDirective Sub-task Resolved Colin McCabe
          33.
          		 Allow LightWeightGSet#Iterator to remove elements Sub-task Resolved Colin McCabe
          34.
          		 recaching improvements Sub-task Closed Colin McCabe
          35.
          		 In CacheReport, don't send genstamp and length on the wire Sub-task Resolved Colin McCabe
          36.
          		 fix broken caching unit tests Sub-task Resolved Andrew Wang
          37.
          		 Loading fsimage fails to find cache pools during namenode startup. Sub-task Resolved Chris Nauroth
          38.
          		 Concurrent clients that add a cache directive on the same path may prematurely uncache from each other. Sub-task Resolved Chris Nauroth
          39.
          		 fix race conditions in DN caching and uncaching Sub-task Closed Colin McCabe
          40.
          		 Add feature documentation for datanode caching. Sub-task Closed Colin McCabe
          41.
          		 Caching RPCs are AtMostOnce, but do not persist client ID and call ID to edit log. Sub-task Resolved Chris Nauroth
          42.
          		 Resolve regressions in Windows compatibility on HDFS-4949 branch. Sub-task Resolved Chris Nauroth
          43.
          		 Fix possible RetryCache hang for caching RPC handlers in FSNamesystem Sub-task Resolved Andrew Wang
          44.
          		 Fixup test-patch.sh warnings on HDFS-4949 branch Sub-task Resolved Andrew Wang
          45.
          		 Support TTL on CacheDirectives Sub-task Closed Andrew Wang
          46.
          		 support cachepool-based limit management in path-based caching Sub-task Closed Andrew Wang
          47.
          		 better API for getting the cached blocks locations Sub-task Closed Andrew Wang
          48.
          		 Add byte and file statistics to PathBasedCacheEntry Sub-task Closed Colin McCabe
          49.
          		 Consistent naming of user-visible caching classes and methods Sub-task Closed Colin McCabe
          50.
          		 add command-line support for modifyDirective Sub-task Resolved Colin McCabe
          51.
          		 Consider maximum DN memory, stale status when scheduling recaching Sub-task Resolved Colin McCabe
          52.
          		 TestPathBasedCacheRequests#testReplicationFactor is flaky Sub-task Closed Andrew Wang
          53.
          		 improve CacheManipulator interface to allow better unit testing Sub-task Closed Colin McCabe
          54.
          		 loading cache path directives from edit log doesn't update nextEntryId Sub-task Closed Colin McCabe
          55.
          		 skip checksums when reading a cached block via non-local reads Sub-task Resolved Colin McCabe
          56.
          		 fix narrow race condition in TestPathBasedCacheRequests Sub-task Closed Colin McCabe
          57.
          		 Rename "path.based" caching configuration options Sub-task Resolved Andrew Wang
          58.
          		 add some more NameNode cache statistics, cache pool stats Sub-task Closed Colin McCabe
          59.
          		 Refactor tests in TestCacheDirectives Sub-task Resolved Andrew Wang
          60.
          		 CacheAdmin help should match against non-dashed commands Sub-task Closed Andrew Wang
          61.
          		 Namenode loops caching and uncaching when data should be uncached Sub-task Closed Andrew Wang
          62.
          		 Hook up cache directive and pool usage statistics Sub-task Closed Andrew Wang
          63.
          		 allow BlockReaderLocal to switch between checksumming and not Sub-task Closed Colin McCabe
          64.
          		 Enforce a max TTL per cache pool Sub-task Closed Andrew Wang
          65.
          		 Remove dfs.namenode.caching.enabled and improve CRM locking Sub-task Closed Colin McCabe
          66.
          		 The CacheManager throws a NPE in the DataNode logs when processing cache reports that refer to a block not known to the BlockManager Sub-task Closed Colin McCabe
          67.
          		 Add per-cache-pool default replication num configuration Sub-task Resolved xupeng

          Activity

            People

              andrew.wang Andrew Wang
              andrew.wang Andrew Wang
              Votes:
              0 Vote for this issue
              Watchers:
              102 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: