[HBASE-24749] Direct insert HFiles and Persist in-memory HFile tracking

• Type: Umbrella
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Affects Version/s: 3.0.0-alpha-1
• Component/s: Compaction, HFile
• Labels: design discussion objectstore storeFile storeengine
• Assignee: Tak-Lon (Stephen) Wu
• Reporter: Tak-Lon (Stephen) Wu
• Created: 21/Jul/20 05:22
• Updated: 23/Jun/22 20:27
• Resolved: 24/Dec/21 02:07

Description

We propose a new feature (a new store engine) to remove the .tmp directory used in the commit stage for common HFile operations such as flush and compaction to improve the write throughput and latency on object stores. Specifically for S3 filesystems, this will also mitigate read-after-write inconsistencies caused by immediate HFiles validation after moving the HFile(s) to data directory.

Please see attached for this proposal and the initial result captured with 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN, and workload C RUN result.

The goal of this JIRA is to discuss with the community if the proposed improvement on the object stores use case makes senses and if we miss anything should be included.

Improvement Highlights
1. Lower write latency, especially the p99+
2. Higher write throughput on flush and compaction
3. Lower MTTR on region (re)open or assignment
4. Remove consistent check dependencies (e.g. DynamoDB) supported by file system implementation

Issue Links

• is related to: HBASE-20431 Store commit transaction for filesystems that do not support an atomic rename
• relates to: HBASE-14090 Redo FS layout; let go of tables/regions/stores directory hierarchy in DFS
• relates to: HBASE-25250 Fixing hacky test in TestCoprocessorInterface
• relates to: HBASE-25249 Adding StoreContext
• relates to: HBASE-26067 Change the way on how we track store file list
• supercedes: HBASE-20431 Store commit transaction for filesystems that do not support an atomic rename
• links to: Direct insert HFiles and Persist in-memory HFile tracking with Milestones
• links to: GitHub Pull Request #3285

Sub-Tasks

• Introduce PersistedStoreEngine and PersistedStoreFileManager: Closed
• Move StoreFlushContext out of HStore and make it pluggable: Closed
• Flush directly into data directory, skip rename when committing flush: Closed
• Direct insert compacted HFiles into data directory.: Closed
• Support split and merge region with direct insert into CF directory: Closed
• Support Snapshot related operation with direct insert HFiles into data/CF directory: Closed
• Integration with direct insert into data/cf directory: Closed
• Migrate hbase:storefile system table into editable meta: Closed
• Cleanup unused/unneeded temporarily HFiles: Closed
• Write HBCK tools to rebuild the tracking: Closed