[HDFS-4949] Centralized cache management in HDFS

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0, 3.0.0-alpha1
• Fix Version/s: 2.3.0
• Component/s: datanode, namenode
• Assignee: Andrew Wang
• Reporter: Andrew Wang
• Created: 02/Jul/13 20:40
• Updated: 04/Jul/22 12:49
• Resolved: 22/Jan/14 22:02

Description

HDFS currently has no support for managing or exposing in-memory caches at datanodes. This makes it harder for higher level application frameworks like Hive, Pig, and Impala to effectively use cluster memory, because they cannot explicitly cache important datasets or place their tasks for memory locality.

Issue Links

• is related to: HDFS-9957 HDFS's use of mlock() is not portable
• is related to: HDFS-5203 Concurrent clients that add a cache directive on the same path may prematurely uncache from each other.
• is related to: HDFS-5385 Caching RPCs are AtMostOnce, but do not persist client ID and call ID to edit log.
• is related to: HDFS-5388 Loading fsimage fails to find cache pools during namenode startup.
• is related to: HDFS-5195 Prevent passing null pointer to mlock and munlock.
• is related to: HDFS-5266 ElasticByteBufferPool#Key does not implement equals.
• is related to: HDFS-5269 Attempting to remove a cache directive fails with NullPointerException.
• is related to: HDFS-5313 NameNode hangs during startup trying to apply OP_ADD_PATH_BASED_CACHE_DIRECTIVE.
• is related to: HDFS-5373 hdfs cacheadmin -addDirective short usage does not mention -replication parameter.
• is related to: YARN-1488 Allow containers to delegate resources to another container
• is related to: HDFS-4953 enable HDFS local reads via mmap
• is related to: HDFS-5197 Document dfs.cachereport.intervalMsec in hdfs-default.xml.
• relates to: HIVE-6347 ZeroCopy read path for ORC RecordReader
• relates to: HDFS-2832 Enable support for heterogeneous storages in HDFS - DN as a collection of storages
• relates to: HDFS-16650 Optimize the cost of obtaining timestamps in Centralized cache management
• relates to: HDFS-5202 Support Centralized Cache Management on Windows.