[IMPALA-4870] Extend Catalog metrics

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: Impala 2.6.0
• Component/s: Catalog
• Labels: catalog-server supportability
• Target Version: Impala 2.13.0
• Assignee: Unassigned
• Reporter: Mostafa Mokhtar
• Created: 02/Feb/17 18:44
• Updated: 02/Nov/18 22:31
• Resolved: 02/Nov/18 22:31

Description

It is difficult to tune and debug the Catalog performance as it doesn't expose any metrics indicating the current state.

The Catalog should expose object count and estimate memory consumed by each object type

• Partitions
• Files
• Blocks

A breakdown per DB/Table/Partition would be ideal.

Also a measure of heap usage by the JVM along with GC time as a % of the JVM lifetime.

Sub-Tasks

• Add JVM heap and non-heap usage in memory metrics and UI: Resolved
• Add JVM thread stacktraces and synchronization info in web UI: Resolved
• Expose per table partition/files/blocks count in web UI: Resolved
• Add average & max wait time metadata loading wait time to the Catalog metrics: Resolved
• Add metrics for tables with missing or incomplete stats: Resolved
• Extend Catalog metrics to list tables that are being loaded along with the ones in the queue: Resolved