[HIVE-25022] Metric about incomplete compactions

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Labels: pull-request-available
• Assignee: Karen Coppage
• Reporter: Karen Coppage
• Created: 15/Apr/21 16:01
• Updated: 21/Apr/21 12:28
• Resolved: 21/Apr/21 12:28

Description

"Compactions in a state" metrics (for example compaction_num_working) count the sum of tables/partitions where the last compaction is in that state.

I propose introducing a new metric about incomplete compactions: i.e. the number of tables/partitions where the last finished compaction* is unsuccessful (failed or "did not initiate"), or where major compaction was unsuccessful then minor compaction succeeded (compaction is not "complete" since major compaction has not succeeded in the time since it should have run).

Example:

These compactions ran on a partition:
major succeeded
major working
major failed
major initiated
major working
major failed
major initiated
major working

The "compactions in a state" metrics will consider the state of this table: working.
The "incomplete compactions" metric will consider this: incomplete, since there have been failed compactions since the last succeeded compaction.

Another example:

These compactions ran on a partition:
major succeeded
major failed
minor failed
minor succeeded

The "compactions in a state" metrics will consider the state of this table: succeeded.
The "incomplete compactions" metric will consider this: incomplete, since there hasn't been a major succeeded since major failed.

Last example:

These compactions ran on a partition:
major succeeded
minor did not initiate

The "compactions in a state" metrics will consider the state of this table: did not initiate.
The "incomplete compactions" metric will consider this: incomplete, since the last compaction was "did not initiate"

*finished compaction: state in (succeeded, failed, attempted/did not initiate)

Issue Links

• links to: GitHub Pull Request #2184