[HIVE-24824] Define metrics for compaction observability

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Labels: pull-request-available
• Assignee: Denys Kuzmenko
• Reporter: Peter Varga
• Created: 24/Feb/21 17:44
• Updated: 23/Mar/21 07:07

Description

Many times if there are failures in the Compaction background processes (Initiator, Worker, Cleaner) it is hard notice the problem until it causes serious performance degradation.
We should create new JMX metrics, that would make it easier to monitor the compaction health. Examples are:

• number of failed / initiated compaction
• number of aborted txns, oldest aborted txns
• tables with disabled compactions and writes
• Initiator and Cleaner cycle runtime
• Size of ACID metadata tables that should have ~ constant rows (txn_to_writeId, completed_txns)

 

Issue Links

• links to: GitHub Pull Request #2016

Sub-Tasks

• Create AcidMetricsService: Closed
• Initiator / Cleaner performance metrics: Resolved
• Worker performance metric: Resolved
• Create new metrics about ACID metadata size: Resolved
• Add host and version information to compection queue: Resolved
• Create new metrics about open transactions: Resolved
• Create new metric about Initiator / Worker hosts: Resolved
• New metrics about aborted transactions: Closed
• Create new metrics about locks: Resolved
• Rename metrics that have spaces in the name: Closed
• Create new metrics about Initiator / Cleaner failures: Closed
• Divide oldest_open_txn into oldest replication and non-replication transactions: Closed
• Metric about incomplete compactions: Resolved
• Add timeout for failed and did not initiate compaction cleanup: Closed
• Create new metrics about the number of delta files in the ACID table: Resolved
• Create metric: Number of tables with > x aborts: Closed
• Put metrics collection behind a feature flag: Closed
• tables_with_x_aborted_transactions should count partition/unpartitioned tables: Closed
• Create metric about oldest entry in "ready for cleaning" state: Closed
• Tweak delta metrics with custom MBean for Prometheus: Closed
• All new compaction metrics should be lower case: Closed
• Number of initiator hosts metric should ignore manually initiated compactions: Closed
• Skip metrics collection about writes to tables with tblproperty no_auto_compaction=true if CTAS: Closed
• Changes to metastore API in HIVE-24880 are not backwards compatible: Closed
• Metrics compaction_failed_initiator_ratio and compaction_failed_cleaner_ratio should be counters: Closed
• Delta metrics collection may cause number of tez counters to exceed tez.counters.max limit: Closed
• Delta metrics keys should contain database name: Closed
• Delta metrics collection may cause NPE: Resolved