[HDFS-7343] HDFS smart storage management

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Wei Zhou
• Reporter: Kai Zheng
• Created: 04/Nov/14 06:02
• Updated: 28/Jun/23 09:06

Description

As discussed in HDFS-7285, it would be better to have a comprehensive and flexible storage policy engine considering file attributes, metadata, data temperature, storage type, EC codec, available hardware capabilities, user/application preference and etc.

Modified the title for re-purpose.
We'd extend this effort some bit and aim to work on a comprehensive solution to provide smart storage management service in order for convenient, intelligent and effective utilizing of erasure coding or replicas, HDFS cache facility, HSM offering, and all kinds of tools (balancer, mover, disk balancer and so on) in a large cluster.

Issue Links

• relates to: HADOOP-13633 Introduce Apache Kafka as a Service into Hadoop
• relates to: HDFS-4672 Support tiered storage policies
• relates to: HDFS-7285 Erasure Coding Support inside HDFS
• relates to: HDFS-2832 Enable support for heterogeneous storages in HDFS - DN as a collection of storages