[HBASE-11339] HBase MOB

• Type: Umbrella
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: hbase-11339, 2.0.0
• Component/s: regionserver, Scanners
• Hadoop Flags: Reviewed

Release Note

The Moderate Object Storage (MOB) feature (HBASE-11339[1]) is modified I/O and compaction path that allows individual moderately sized values (100KB-10MB) to be stored in a way that write amplification is reduced when compared to the normal I/O path. MOB is defined in the column family and it is almost isolated with other components, the features and performance cannot be effected in normal columns.

For more details on how to use the feature please consult the HBase Reference Guide

• Assignee: Jingcheng Du
• Reporter: Jingcheng Du
• Created: 13/Jun/14 07:30
• Updated: 17/Jun/22 08:53
• Resolved: 22/Jul/15 23:13

Description

It's quite useful to save the medium binary data like images, documents into Apache HBase. Unfortunately directly saving the binary MOB(medium object) to HBase leads to a worse performance since the frequent split and compaction.
In this design, the MOB data are stored in an more efficient way, which keeps a high write/read performance and guarantees the data consistency in Apache HBase.

Issue Links

• breaks: HBASE-12418 Multiple branches for MOB feature breaking git for case insensitive filesystems
• depends upon: HBASE-11591 Scanner fails to retrieve KV from bulk loaded file with highest sequence id than the cell's mvcc in a non-bulk loaded file
• relates to: HBASE-22749 Distributed MOB compactions

Sub-Tasks

• Read and write MOB in HBase: Closed
• External MOB compaction tools: Closed
• Handle the MOB in compaction: Closed
• MOB integration testing: Closed
• Snapshot for MOB: Closed
• Metrics for MOB: Closed
• Native MOB Compaction mechanisms.: Closed
• Improve the value size of the reference cell in mob column: Closed
• Document MOB in Ref Guide: Closed
• isMob and mobThreshold do not follow column descriptor property naming conventions: Closed
• Avoid major compaction in TestMobSweeper: Closed
• Shorten the run time of integration test by default when using mvn failsafe:integration-test: Closed
• mob status should print human readable numbers.: Closed
• Support the mob attributes in hbase shell when create/alter table: Closed
• Clean the code after adding IS_MOB and MOB_THRESHOLD to column family: Closed
• Shorten the mob snapshot unit tests: Closed
• [mob] improve how we resolve mobfiles in reads: Closed
• Correct a typo in the mob metrics: Closed
• Incorrect implementation of CompactionRequest.isRetainDeleteMarkers: Closed
• Move the mob table name tag to the 2nd one: Closed
• Explicitly flush the file name in sweep job: Closed
• Incorrect 'mobFileCacheMissCount' calculated in the mob metrics: Closed
• Incorrect log info in the store compaction of mob: Closed
• Ignore the count of mob compaction metrics when there is issue: Closed
• SnapshotInfo tool does not find mob data in snapshots: Closed
• Have compaction scanner save info about delete markers: Closed
• Add unit tests that exercise the added hfilelink link mob paths: Closed
• Add a UT to read mob file when the mob hfile moving from the mob dir to the archive dir: Closed
• sweep job needs to exit non-zero if job fails for any reason.: Closed
• Add mob cell count to the metadata of each mob file: Closed
• treat mob region as any other region when generating rs manifest.: Closed
• Use table lock instead of MobZookeeper: Closed
• Add shell commands to trigger the mob file compactor: Closed
• Add read lock to ExpiredMobFileCleanerChore: Closed
• Refactor MOB Snapshot logic to reduce code duplication.: Closed
• improve mob sweeper javadoc: Closed
• IllegalArgumentException in compaction when table has a namespace: Closed
• NPE in ExpiredMobFileCleanerChore: Closed
• Add support for mob in TestAcidGuarantees and IntegrationTestAcidGuarantees: Closed
• Add mob compaction actions and monkeys to Chaos Monkey: Closed
• add mob_threshold option to load test tool: Closed
• [mob] reads hang when trying to read rows with large mobs (>10MB): Closed
• fix new javadoc warns introduced by mob.: Closed
• Skip the disabled table in mob compaction chore and MasterRpcServices: Closed
• Flakey failures of TestAcidGuarantees#testMobScanAtomicity: Closed
• Mob files are not encrypting in mob compaction and Sweeper: Closed
• Add delay for the first execution of ExpiredMobFileCleanerChore and MobFileCompactorChore: Closed
• Remove KeyValueUtil.ensureKeyValue(cell) from MOB code.: Closed
• Use the same HFileContext with store files in mob files: Closed
• Handle the rename, annotation and typo stuff in MOB: Closed
• Remove the DeleteTableHandler: Closed
• Remove the DeleteTableHandler: Closed
• Disable the MobCompactionChore when the interval is not larger than 0: Closed
• Revert the changes in pom.xml: Closed
• Use LimitInputStream in hbase-common instead of ProtobufUtil.LimitedInputStream: Closed
• Check the mob files when there are mob-enabled columns in HFileCorruptionChecker: Closed
• Do not reset the mvcc for bulk loaded mob reference cells in reading: Closed
• Race in multi threaded PartitionedMobCompactor causes NPE: Closed
• Wrong mob metrics names in TestRegionServerMetrics: Closed
• Return empty value when the mob file is corrupt instead of throwing exceptions: Closed
• Do not reset mvcc in compactions for mob-enabled column: Closed
• Add mob integrity check in HFilePrettyPrinter: Closed
• Not cleaning Mob data when Mob CF is removed from table: Closed