[HIVE-10761] Create codahale-based metrics system for Hive

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0, 2.0.0
• Component/s: Diagnosability
• Assignee: Szehon Ho
• Reporter: Szehon Ho
• Created: 19/May/15 23:15
• Updated: 16/Feb/16 23:50
• Resolved: 04/Jun/15 06:52

Description

There is a current Hive metrics system that hooks up to a JMX reporting, but all its measurements, models are custom.

This is to make another metrics system that will be based on Codahale (ie yammer, dropwizard), which has the following advantage:

• Well-defined metric model for frequently-needed metrics (ie JVM metrics)
• Well-defined measurements for all metrics (ie max, mean, stddev, mean_rate, etc),
• Built-in reporting frameworks like JMX, Console, Log, JSON webserver

It is used for many projects, including several Apache projects like Oozie. Overall, monitoring tools should find it easier to understand these common metric, measurement, reporting models.

The existing metric subsystem will be kept and can be enabled if backward compatibility is desired.

Issue Links

• breaks: HIVE-10944 Fix HS2 for Metrics
• relates to: HIVE-10725 Better resource management in HiveServer2
• links to: Review board

Sub-Tasks

• Create Json-servlet reporter for metrics: Resolved
• Add number of HMS/HS2 connection metrics: Closed
• Add lock metrics to HS2: Closed
• Add metrics around HS2 query execution and job submission for Hive: Closed
• Add HMS metrics for number of tables and partitions: Closed
• Cleanup unused metrics in HMS: Closed
• UX improvements for HIVE-12499: Closed
• Fix empty operation-pool metrics: Closed