[HBASE-20945] HBase JMX - timestamp of last Major Compaction (started, completed successfully)

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.1.2
• Component/s: API, Compaction, master, metrics, monitoring, regionserver, tooling
• Assignee: Unassigned
• Reporter: Hari Sekhon
• Created: 26/Jul/18 09:21
• Updated: 01/Aug/18 08:41

Description

Request that the timestamp of the last major compaction be stored in JMX API available at /jmx.

Major Compactions may be disabled to better control scheduling to trigger off peak (this is an old school recommendation), but there is a risk that the major compaction doesn't happen in that case. Also people may trigger major compactions manually and it's hard to see that (I've looked at graphs of storefile counts where it's not obvious but I can infer it from spikes in compaction queue length). Storing the last timestamps would allow all sorts of scripting checks against the API much more simply than trying to infer it from changes in graphs. Also with recent changes to allow compactions to be cancelled in HBASE-6028, the queue length doesn't tell the whole story as the compaction may not have happened if it got cancelled, so the compaction queue spike will be there even though major compaction did not in fact happen/complete.

Since major compactions may take hours and can also now be cancelled in the latest versions of HBase, we need a few different fields added to JMX:

• HBase Master JMX:
  • timestamp that last major compaction was triggered, either manually via major_compact command or via schedule
  • timestamp that last major compaction completed successfully (since timestamp above could have been started and then later cancelled manually if load was too high)
• HBase Regionserver JMX:
  • timestamp per region that last major compaction was triggered (there are already compcationsCompletedCount, numBytesCompactedCount and numFilesCompactedCount so it makes sense to add this next to those for each region)
  • timestamp per region that last major compaction completed successfully

Issue Links

• is related to: HBASE-20946 HMaster UI & JMX - warn if Major Compaction hasn't occurred after hbase.hregion.majorcompaction time
• is related to: HBASE-20947 HMaster UI - expose JMX timestamps of last Major Compaction
• relates to: HBASE-6028 Implement a cancel for in-progress compactions