[HBASE-22749] Distributed MOB compactions

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.5.0
• Component/s: mob

Release Note

<!-- markdown -->
MOB compaction is now handled in-line with per-region compaction on region
  servers

- regions with mob data store per-hfile metadata about which mob hfiles are
  referenced
- admin requested major compaction will also rewrite MOB files; periodic RS
  initiated major compaction will not
- periodically a chore in the master will initiate a major compaction that
  will rewrite MOB values to ensure it happens. controlled by
  'hbase.mob.compaction.chore.period'. default is weekly
- control how many RS the chore requests major compaction on in parallel
  with 'hbase.mob.major.compaction.region.batch.size'. default is as
  parallel as possible.
- periodic chore in master will scan backing hfiles from regions to get the
  set of referenced mob hfiles and archive those that are no longer
  referenced. control period with 'hbase.master.mob.cleaner.period'
- Optionally, RS that are compacting mob files can limit write
  amplification by not rewriting values from mob hfiles over a certain size
  limit. opt-in by setting 'hbase.mob.compaction.type' to 'optimized'.
  control threshold by 'hbase.mob.compactions.max.file.size'.
  default is 1GiB
- Should smoothly integrate with existing MOB users via rolling upgrade.
  will delay old MOB file cleanup until per-region compaction has managed
  to compact each region at least once so that used mob hfile metadata can
  be gathered.

This improvement obviates the dataloss in HBASE-22075.

• Assignee: Vladimir Rodionov
• Reporter: Vladimir Rodionov
• Created: 27/Jul/19 02:26
• Updated: 14/Jul/22 19:50
• Resolved: 19/Apr/20 02:50

Description

There are several drawbacks in the original MOB 1.0 (Moderate Object Storage) implementation, which can limit the adoption of the MOB feature:

1. MOB compactions are executed in a Master as a chore, which limits scalability because all I/O goes through a single HBase Master server.
2. Yarn/Mapreduce framework is required to run MOB compactions in a scalable way, but this won’t work in a stand-alone HBase cluster.
3. Two separate compactors for MOB and for regular store files and their interactions can result in a data loss (see HBASE-22075)

The design goals for MOB 2.0 were to provide 100% MOB 1.0 - compatible implementation, which is free of the above drawbacks and can be used as a drop in replacement in existing MOB deployments. So, these are design goals of a MOB 2.0:

1. Make MOB compactions scalable without relying on Yarn/Mapreduce framework
2. Provide unified compactor for both MOB and regular store files
3. Make it more robust especially w.r.t. to data losses.
4. Simplify and reduce the overall MOB code.
5. Provide 100% compatible implementation with MOB 1.0.
6. No migration of data should be required between MOB 1.0 and MOB 2.0 - just software upgrade.

Issue Links

• causes: HBASE-25166 MobFileCompactionChore is closing the master's shared cluster connection
• is related to: HBASE-22075 Potential data loss when MOB compaction fails
• is related to: HBASE-11339 HBase MOB
• relates to: HBASE-23571 Handle CompactType.MOB correctly
• relates to: HBASE-23363 MobCompactionChore takes a long time to complete once job
• relates to: HBASE-23724 Change code in StoreFileInfo to use regex matcher for mob files.
• links to: GitHub Pull Request #623
• links to: GitHub Pull Request #921
• links to: GitHub Pull Request #4581

Sub-Tasks

• MobFileCleanerChore test case: Resolved
• Finalize I/O optimized MOB compaction: Resolved
• Convert MobCompactionTest into integration test: Resolved
• Documentation and release notes: Resolved
• Simplify logic in DefaultMobStoreCompactor: Resolved
• Test case for MOB compaction in a region's batch mode.: Resolved
• Test case for MOB compaction in a regular mode.: Resolved
• Ensure MOB compaction works in optimized mode after snapshot clone: Resolved
• Revert optimized IO back to general compaction during upgrade/migration process: Resolved
• Correct snapshot handling: Resolved