[HBASE-20429] Support for mixed or write-heavy workloads on non-HDFS filesystems

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Labels: s3
• Assignee: Unassigned
• Reporter: Andrew Kyle Purtell
• Created: 16/Apr/18 20:25
• Updated: 22/Apr/23 21:52
• Resolved: 22/Apr/23 21:52

Description

We can support reasonably well use cases on non-HDFS filesystems, like S3, where an external writer has loaded (and continues to load) HFiles via the bulk load mechanism, and then we serve out a read only workload at the HBase API.

Mixed workloads or write-heavy workloads won't fare as well. In fact, data loss seems certain. It will depend in the specific filesystem, but all of the S3 backed Hadoop filesystems suffer from a couple of obvious problems, notably a lack of atomic rename.

This umbrella will serve to collect some related ideas for consideration.

Issue Links

• relates to: HBASE-21070 SnapshotFileCache won't update for snapshots stored in S3

Sub-Tasks

• Store commit transaction for filesystems that do not support an atomic rename: Closed