[HBASE-6572] Tiered HFile storage

• Type: Brainstorming
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Assignee: Unassigned
• Reporter: Andrew Kyle Purtell
• Created: 13/Aug/12 21:31
• Updated: 13/Jun/22 19:06
• Resolved: 11/Jun/22 18:10

Description

Consider how we might enable tiered HFile storage. If HDFS has the capability, we could create certain files on solid state devices where they might be frequently accessed, especially for random reads; and others (and by default) on spinning media as before. We could support the move of frequently read HFiles from spinning media to solid state. We already have CF statistics for this, would only need to add requisite admin interface; could even consider an autotiering option.

Dhruba Borthakur did some early work in this area and wrote up his findings: http://hadoopblog.blogspot.com/2012/05/hadoop-and-solid-state-drives.html . It is important to note the findings but I suggest most of the recommendations are out of scope of this JIRA. This JIRA seeks to find an initial use case that produces a reasonable benefit, and serves as a testbed for further improvements. If I may paraphrase Dhruba's findings (any misstatements and errors are mine): First, the DFSClient code paths introduce significant latency, so the HDFS client (and presumably the DataNode, as the next bottleneck) will need significant work to knock that down. Need to investigate optimized (perhaps read-only) DFS clients, server side read and caching strategies. Second, RegionServers are heavily threaded and this imposes a lot of monitor contention and context switching cost. Need to investigate reducing the number of threads in a RegionServer, nonblocking IO and RPC.

Issue Links

• depends upon: HDFS-3672 Expose disk-location information for blocks to enable better scheduling
• duplicates: HBASE-12935 Does any one consider the performance of HBase on SSD?
• is related to: HDFS-6875 Archival Storage: support migration for a list of specified paths
• is related to: HDFS-4672 Support tiered storage policies
• is related to: HBASE-4755 HBase based block placement in DFS
• is related to: HBASE-7404 Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE
• is related to: HDFS-2832 Enable support for heterogeneous storages in HDFS - DN as a collection of storages
• is related to: HDFS-5682 Heterogeneous Storage phase 2 - APIs to expose Storage Types
• is related to: HDFS-6584 Support Archival Storage
• is related to: HBASE-6799 Store more metadata in HFiles
• relates to: HBASE-18118 Default storage policy if not configured cannot be "NONE"
• relates to: HDFS-4672 Support tiered storage policies
• relates to: HDFS-3672 Expose disk-location information for blocks to enable better scheduling

Sub-Tasks

• Utilize Flash storage for WAL: Closed
• Utilize Flash storage for flushing: Closed
• Add more docs and a basic check for storage policy handling: Closed
• Support CF-level Storage Policy: Closed
• Support setting storage policy in bulkload: Closed
• Add document about Heterogeneous Storage Management (HSM) in hbase: Open